[postgis-users] shp2pgsql fails on malformed integer attribute

[Aman Verma]

Hi,

I understand why would you be reluctant to do automatic correction. However, the shapelib library, which shp2pgsql directly uses, does do automatic correction.

I agree: any mistake in the dbf should be reported. However, I believe that, at the very least, a switch should be supported that allows automatic conversion.

I don't quite understand your example :

"if the field numeric in the shapefile instead of be fill with numbers was filled with '1', '2', '3', etc...
where the number are however not really numbers but text based, an action of importing always 0.
Will be more confusing. The user don't understand why a number (or simil-number) became a '0'."

In the DBF file itself, there is no difference between a 'text-based number' and normal numbers. Whether I declare the column as numeric or text, the data for the number one (1) will look the same. That will never be an issue.

It is interesting that you bring up other examples of corrupt DBFs: I believe that shapelib actually handles some of those examples.

I agree with you that nothing should be 'hidden'. I think the behaviour should conform to shapelib, which it already mostly does. At the very least, the DBF should be checked for this kind of corruption, and reported.

aman

From: postgis-users-bounces at postgis.refractions.net [mailto:postgis-users-bounces at postgis.refractions.net] On Behalf Of Andrea Peri
Sent: August 14, 2010 3:18 AM
To: PostGIS Users Discussion
Subject: [postgis-users] shp2pgsql fails on malformed integer attribute

>  aman.verma at mcgill.ca<http://mcgill.ca>  wrote..
>If you use shp2pgsql to upload a shapefile with an attribute table that has a malformed integer
>(like a letter), shp2pgsql will not correct it, and the upload will fail.
>Normally, I would view this as 'expected behaviour' (garbage in, garbage out). However, I have two
>reasons to believe that shp2pgsql should do the correction:
>1) ArcGIS, and other software that read the xBase format (DBF), interpret such malformed integers as 0.
>There seem to be a lot of shape files floating around that have malformed integers in them, precisely
>because they appear to be working. Users may have a difficult time understanding why shp2pgsql keeps
>trying to upload 'g' when all they can see is '0'.

I think this is not a good strategy.

If there a mistake in the dbf the mistake must be reported, not hide resolving automatically it.
Otherwise the shapefile version became not-interoperability.
Just now often the gis user was speaken about "shapefile read from arcgis" or some other tools, as they was difference from other kind of shapefile.

I experience many situation where a shapefile send me from other users are wrong, and the user report me "but my tool read it so for me it is  ok."

This may be born many kinds of formats shapefiles difference each other.
So the shapefile format "became an opinion", and every tool apply they specific changing to the format.
For example

what if a filed name was more then 10 character ?
what if a text field was more character of how much declared in the header ?
what if a date field was filled with a number ?
and so on .....

Even in your case:

if the field numeric in the shapefile instead of be fill with numbers was filled with '1', '2', '3', etc...
where the number are however not really numbers but text based, an action of importing always 0.
Will be more confusing. The user don't understand why a number (or simil-number) became a '0'.

Andrea.

--
-----------------
Andrea Peri
. . . . . . . . .
qwerty àèìòù
-----------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20100814/52e8a20c/attachment.html>