[postgis-devel] [postgis-users] PSC Vote: Keep or drop Flatgeobuf in PostGIS 3.2.0
Jeff McKenna
jmckenna at gatewaygeomatics.com
Thu Nov 25 09:07:55 PST 2021
As an aside, I appreciate this explanation on speed and benefits, Björn,
of FlatGeobuf. Thanks,
-jeff
--
Jeff McKenna
GatewayGeo: Developers of MS4W, MapServer Consulting and Training
co-founder of FOSS4G
http://gatewaygeo.com/
On 2021-11-25 7:32 a.m., Björn Harrtell wrote:
> Just a FWIW,
>
> FlatGeobuf is actually not prioritizing space optimization, instead
> designed for simplicity and speed. Even so, it can be smaller than both
> shapefile and geojson but it depends on data. But so, it can make sense
> to compress if space optimization is required.
>
> The primary reason for why FlatGeobuf is fast is the use of Flatbuffers
> which is not far from C structs (i.e direct access, no decode, no extra
> allocations needed) but has a stable platform independent wire
> format. If created without index FlatGeobuf can be both written and read
> as stream, which makes it very fast as a system to system transfer
> format. If written with an index a FlatGeobuf can be accessed with a
> bbox filter very efficiently because it has a guaranteed balanced
> Hilbert R-tree, so can be traversed with forward seek and found data
> locality is good because data is ordered by the spatial index.
> Unfortunately bytea input/output does not allow full advantage of these
> properties - something like COPY would be better (since it can be streamed).
>
> /Björn
>
> Den tors 25 nov. 2021 kl 00:10 skrev Regina Obe <lr at pcorp.us
> <mailto:lr at pcorp.us>>:
>
> As mentioned on IRC just reiterated here. ____
>
> __ __
>
> I would like to see ST_AsFlatGeoBuf in there. Here are my reasons.____
>
> __ __
>
> 1) To Bruce’s point that sure ogr_fdw can read it, I’m more
> interested in the writing of it (which at least I can’t do from
> scratch with ogr_fdw) and getting out of the database onto the
> command line is not one of my great ambitions in life.____
>
> __ __
>
> 2) The fact it is a lighter weight format than ST_AsGeoJSON which
> has the same 1GB limitation.____
>
> Or as mentioned https://flatgeobuf.org/ <https://flatgeobuf.org/> a
> better shapefile format (for < 1GB)____
>
> And the fact that FlatGeobuf is about 50% lighter than GeoJSON ____
>
> From site:____
>
> (shapefile 1, FlatGeoBuf 0.77, GeoJSON 1.2)____
>
> __ __
>
> Means I can stuff much more data in the 1GB limit than I can in
> GeoJSON.____
>
> Though not sure about how that places with the other attribute data
> if they are compacted as efficiently.____
>
> __ __
>
> 3) Given it is demonstrated it can work fine with leaflet and
> openlayers means it is possible to hook it into something like ____
>
> https://github.com/CrunchyData/pg_featureserv
> <https://github.com/CrunchyData/pg_featureserv> ____
>
> __ __
>
> as an alternative format to geojson (not saying you should Martin
> Davis – just throwing it out there J)____
>
> __ __
>
> 4) One more reason to fight that 1GB limit – as postgis raster is
> impacted by that too.____
>
> __ __
>
> __ __
>
> Thanks,____
>
> Regina____
>
> __ __
>
> *From:*postgis-users [mailto:postgis-users-bounces at lists.osgeo.org
> <mailto:postgis-users-bounces at lists.osgeo.org>] *On Behalf Of *Björn
> Harrtell
> *Sent:* Wednesday, November 24, 2021 5:26 PM
> *To:* PostGIS Development Discussion <postgis-devel at lists.osgeo.org
> <mailto:postgis-devel at lists.osgeo.org>>
> *Cc:* PostGIS Users Discussion <postgis-users at lists.osgeo.org
> <mailto:postgis-users at lists.osgeo.org>>
> *Subject:* Re: [postgis-users] [postgis-devel] PSC Vote: Keep or
> drop Flatgeobuf in PostGIS 3.2.0____
>
> __ __
>
> Even though I spent quite a bit of effort on implementing this stuff
> and I'm sure I can fix the crashers I agree with the arguments to
> remove it. That is, 1GB limit is really bad and better to use GDAL
> which has a well maintained impl of it.____
>
> __ __
>
> If there was a way to stream in and out binary with custom encoding
> and no size limit (i.e COPY with custom/ext binary format) it could
> make sense but I don't think that is going to happen any time soon.____
>
> __ __
>
> Oh well, it was fun. Some of it. 😂____
>
> __ __
>
> PS. ST_AsGeobuf should be deprecated/removed too - it's even less
> useful IMHO.____
>
> __ __
>
> PS2. I do still believe in FlatGeobuf and it is used in production.
> ;)____
>
> __ __
>
> /Björn____
>
> Den ons 24 nov. 2021 22:26Bruce Rindahl <bruce.rindahl at gmail.com
> <mailto:bruce.rindahl at gmail.com>> skrev:____
>
> FWIW I say remove it and seriously think about not including it
> at all. Looks like you can use the format right now via ogr_fdw
> using GDAL. ____
>
> __ __
>
> On Wed, Nov 24, 2021 at 12:51 PM Regina Obe <lr at pcorp.us
> <mailto:lr at pcorp.us>> wrote:____
>
> FWIW it’s already in GDAL since 3.1 and yah GDAL is a better
> home since it doesn’t have the 1GB PostgreSQL limitation____
>
> ____
>
> https://gdal.org/drivers/vector/flatgeobuf.html
> <https://gdal.org/drivers/vector/flatgeobuf.html>____
>
> ____
>
> Also here are OpenLayers and Leaflet examples for those not
> familiar with the format____
>
> ____
>
> OpenLayers: https://flatgeobuf.org/examples/openlayers/
> <https://flatgeobuf.org/examples/openlayers/>____
>
> ____
>
> ____
>
> Leaflet: https://flatgeobuf.org/examples/leaflet/
> <https://flatgeobuf.org/examples/leaflet/>____
>
> ____
>
> Thanks,____
>
> Regina____
>
> ____
>
> *From:*postgis-users
> [mailto:postgis-users-bounces at lists.osgeo.org
> <mailto:postgis-users-bounces at lists.osgeo.org>] *On Behalf
> Of *Darafei "Kom?pa" Praliaskouski
> *Sent:* Wednesday, November 24, 2021 3:27 PM
> *To:* PostGIS Development Discussion
> <postgis-devel at lists.osgeo.org
> <mailto:postgis-devel at lists.osgeo.org>>
> *Cc:* PostGIS Users Discussion
> <postgis-users at lists.osgeo.org
> <mailto:postgis-users at lists.osgeo.org>>
> *Subject:* Re: [postgis-users] [postgis-devel] PSC Vote:
> Keep or drop Flatgeobuf in PostGIS 3.2.0____
>
> ____
>
> Hi,
>
> I have not seen flatgeobuf in the wild, and I believe it can
> be safely removed. ____
>
> ____
>
> The current implementation is impaired by Postgres' life
> choices of 1GB limit and thus not usable for any data, just
> size-limited subset. ogr2ogr seems like a better suited
> place for it to reside.____
>
> ____
>
> I'm -0 on adding flatgeobuf to core, and -1 on releasing
> with known crashers. This would converge to "remove if
> nobody can fix crashers".____
>
> ____
>
> ____
>
> On Wed, Nov 24, 2021 at 11:10 PM Regina Obe <lr at pcorp.us
> <mailto:lr at pcorp.us>> wrote:____
>
> This is a PSC vote, but we would like some feedback on
> this from packagers
> and users as such comments will sway our vote.
>
> We have two blockers that center around the new
> FlatGeoBuf format.
>
> https://trac.osgeo.org/postgis/ticket/5005
> <https://trac.osgeo.org/postgis/ticket/5005> (this one
> is easily
> replicatable)
>
> https://trac.osgeo.org/postgis/ticket/5014
> <https://trac.osgeo.org/postgis/ticket/5014> (this one I
> can only replicate
> with the cowbuilder setup Bas Cowenberg provided)
>
> both I think are manifestations of the same problem how
> the header is
> derived and what it's doing with numeric and geometry
> fields.
>
> I've taken a stab at troubleshooting and fixing, but did
> not have much luck.
> That said, if anyone is willing to help fix that would
> be great and fix
> within a 1 to 2 week time period.
>
> If not I feel that we really need to take it out of our
> PostGIS 3.2.0
> release (which will be going on to 3.2.0beta2).
>
> I'd like to release PostGIS 3.2.0beta2 in about a week
> or so with flatgeobuf
> fixed or removed. If removed, we'll push flatgeobuf to
> PostGIS 3.3.0
> cycle.
>
> Thanks,
> Regina
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> <mailto:postgis-devel at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
> <https://lists.osgeo.org/mailman/listinfo/postgis-devel>____
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> <mailto:postgis-devel at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
> <https://lists.osgeo.org/mailman/listinfo/postgis-devel>____
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org <mailto:postgis-devel at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
> <https://lists.osgeo.org/mailman/listinfo/postgis-devel>____
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org <mailto:postgis-devel at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
> <https://lists.osgeo.org/mailman/listinfo/postgis-devel>
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
More information about the postgis-devel
mailing list