Welcome to Defra Data Services Forum

    Lidar data - ASCII to GeoTiffs

    Follow

    We have just released our latest 1m and 2m DTM Composite products on the DEFRA DSP. This includes the best available surveys captured before October 2019. With the 2019 release we have changed our data format from ASCII to GeoTiffs. These GeoTIff files are supplied in 5km tiles with elevations in metres. GeoTIFF is a modern GIS format widely used for delivering elevation data and was chosen due to its open standards and highly efficient data storage. Data volumes are on average 5x smaller, enabling faster download over internet and more manageable storage. Storing the data in GeoTiff also enables us to start to deliver more services over the web such as Image Services and WCS services. These services provide APIs for users view the data directly in a browser, or bring the data directly into desktop GIS and perform geoprocessing analysis on it. The old ASCII format meant we were limited to what we could offer. Moving to GeoTiff format also aligns the Composite product to our National LIDAR Programme products.

     The vast majority of software’s including QGIS, ArcGIS, GRASS & AutoCAD Civil3D are able to open a GeoTiff file in the same way as the previous ASCII file. If you do experience any issues we have put together some links on converting between file formats on the Blogs and Guides section of our storymap. This is a straightforward conversion process in GIS software or can be achieved with a few lines of code.

    Was this article helpful?
    3 out of 3 found this helpful

    Comments

    Please sign in to leave a comment

    • Avatar
      Ian Williams

      Hi Ella,
      This change is causing me a real headache. The old ASCII files were dead easy to read and required no special software. I don't (yet) know much about the GEOTIFF format, but I can't open a TIF file with a text editor or easily read the data into my own software. This move isn't helping to democratise the data. Assuming I ever manage to figure out a way of reading a GEOTIFF file (without having to resort to a third party application), does the format give me xyz point coordinate values as the good old ASCII files did or is it just some image file?

    • Avatar
      Data Services Platform team

      Hi – It’s not possible to open the GeoTIFF file and extract information using basic windows software. The GeoTIFF file format can be opened in QGIS – A free and open source GIS software package. From this you could easily convert back to ASCII if you have specific software you have written that you want to open it in. The GeoTIFF file can also be read using python gdal libraries to extract the XYZ position of each pixel if you are importing into your own software or convert back into ACSCII.
      The GeoTIFF File provides a XYZ position for the centre point of the pixel to the exact same positional accuracy as the previous ASCII files. As mentioned in the FAQ moving to GeoTIFFS means out LIDAR data can be support more modern web GIS applications as well as being significantly smaller which helps with storage, download and transfer costs.

    • Avatar
      Ian Williams

      Hi Ella,
      Thanks for your reply. I've since figured out how to import the new data format into my Mathematica notebooks. I hear what you say about using third party conversion tools such as QGIS, etc. but would prefer not to if possible as things start to get messy if flipping between several different software applications. I do think it's a shame that you've dropped the ASCII format as it was very accessible. Why not just provide the data in both formats?

    • Avatar
      Data Services Platform team

      Unfortunately the DSP isn’t currently able to support multiple download formats on the open data side for raster data. It would be fantastic if both formats could be offered however this would require both formats to be stored on the system and processed adding significant storage costs and resource time to enable this. As an organisation we have also moved away from using ASCII data and we no longer store newly captured LIDAR data in this format.
      Many thanks
      Ella

    • Avatar
      Peter Todd

      I've been using a script to convert downloaded ASC data to JPEG2000, which results in vastly smaller file-sizes than either ASC or TIFF. It could be 32bit lossless float, but for my purposes I'm using lossy normalised 16bit uint. At some point I'll do a bit more comparitive analysis of different settings (I seem to remember Pillow complained when I tried to write float data). I'm currently using images compressed with 60db SNR (maybe a bit too aggressively lossy for some applications) and they end up being <1.5mb for a 50cm (2000x2000) tile (generally much less than that: many tiles with no void areas clocking in at ~100kb) while still giving a pretty good representation of the data.

      I realise that this format isn't as widely supported as others. There are open geospatial consortium standards for it though, and it should be ok in GIS software. So anyway, it seems to me that if more people got used to using it, there may be great benefits to offering the data in that format. I'd probably stop short of recommending it for definitive archival purposes, though. If there were to be a second choice available, I'd like it to be that. I really want to get hold of the complete set of survey in that format and would be happy to share scripts etc if that helps (I've been meaning to make a request for the data in its existing form, but thought I'd I'll mention it now since I'm here).

      FWIW it looks as though Mathematica also supports it, although the devil is in the details and I'm pretty sure it should support TIFF as well, but vagueries of precise versions can be problematic and I doubt it'd be any easier for Ian (who I'm afraid is probably going to have to learn to adapt). I must concur that I often find TIFF pretty irritating to work with in various ways.

    • Avatar
      Michael Eldred

      Hello Ella,
      My GIS software supports GeoTIFF but when I download lidar data from the DEFRA website and open the .zip I get 8 files none of which are .geoTIFF or anything like it. In case it's not obvious, I know virtually nothing about the workings of GIS and am currently at a loss to know how else to get the data that used to work so easily with .asc files. Can you help please?

    • Avatar
      Michael Eldred

      Hello Ella
      Problem sorted no need to respond

    • Avatar
      Data Services Platform team

      Thats great, thanks for letting me know

    • Avatar
      Richard Newell (Senior Asset Engineer)

      Michael. I needed to know the answer to that question as I haven't worked it out.
      Hi Ella, How do I use the 8 files in the .zip folder, please?
      Do you know how I can use the LIDAR data in a HEC-RAS software to analyse river flood problems, please?
      Thanks.
      Richard

    • Avatar
      Data Services Platform team

      Hi Richard,
      These files make up the shapefile metadata index catalogue for the downloaded 5km GeoTIFF surface model which tells you information about when the data was flown etc. A shapefile is made up of the various components, hence why there are 8 files. When you open the shapefile in a GIS it will likely only show a single file (usually the .shp) which you add to your map.

      The actual surface model GeoTIFF is in the root folder of the zip file. The GeoTIFF is made up of 3 files:
      - *.tif – the surface model
      - *.tfw – the world file telling it where it resides
      - *.tif.xml – xml metadata which when opened in supported software provides metadata information about the GeoTIFF such as when the data was flown. This information is also in the index shapefile.

      I’m not able to help with the software question as I’ve not heard of it I’m afraid – can you go to your suppliers and ask for advice.

      I hope this helps

    • Avatar
      Rose Jackson

      Hi,
      Are DSM Composite data also available in GeoTIFF format or is only DTM currently accessible?
      Many thanks,
      Rose

    • Avatar
      Data Services Platform team

      Hi Rose,
      Currently on the DSM Composite is only available for the 2017 version in ASCII format, not GeoTIFF,
      We will soon be releasing the 2020 DTM composite, followed by the 2020 DSM composite at 1m and 2m resolutions (mid-summer). At which point we will retire the 2019 and 2017 versions.

      I hope this helps
      Data Sharing & Access team

    • Avatar
      Sandra Kennish

      Hi

      I'm trying to download a large area of DTM - LIDAR Composite DTM 2020 - 1m or the data from the National Lidar Program. I have a shapefile of our area, however as it is a large area I always get an error saying the area is too large. Is there anyway of downloading large areas? (Full National park plus buffer.) Usually we have to get the data sent on a memory stick. Thanks

      Sandra

    • Avatar
      Data Services Platform team
    • Avatar
      Sandra Kennish

      Hi Dan, thank you. I did do that actually, but it seems to have gone in as a FOI, which I seem to get various emails about, I did this on 20 July, but still waiting for the data. It isn't really an FOI as the data is freely available. Maybe I will put the request on again.

      thanks

      Sandra

    • Avatar
      Data Services Platform team

      Sandra,

      Let me look into it for you. Don't raise another request yet.

      Regards,
      Dan, Data Sharing & Access Team