Passing Water Data to R Script via WPS4R

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing Water Data to R Script via WPS4R

Raul
Dear Community,

I work in a project that have real water level sensors and I have an ubuntu 12.04 LTS 64 bits server running SOS v4.1. I have made a Java program that sends a GetObservation request and successfully receives an XML with the sensor data. In that same server I have the R v3.2.0, Rserver v3.2.0, WPS v3.3.1 and WPS4R (with GeoTools v3.3.1) running. I have deployed a simple annotated R script and successfully tested it with the 52n web client (script and request attached). Now I want to make a Java program that sends the sensor data, mainly latitude, longitude and water level, as input parameters to an R script via WPS request and interpolate them in the script, possibly using the gstat library, but I don't know how to do it. So, my questions are:

1. How to request this process? What must be the types of the input and output parameters in the XML?
2. What must be the types of the input and output parameters in the script? How to annotate them?

I will greatly appreciate if you can give examples because I haven't much experience in these technologies.
Thanks in advance.

Best regards,
Raul

request.xml
TestScript.R
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Daniel
Hi Raul,

thanks for your interest in WPS4R!

Am 27/05/2015 um 02:00 schrieb Raul:

> I work in a project that have real water level sensors and I have an ubuntu
> 12.04 LTS 64 bits server running SOS v4.1. I have made a Java program that
> sends a GetObservation request and successfully receives an XML with the
> sensor data. In that same server I have the R v3.2.0, Rserver v3.2.0, WPS
> v3.3.1 and WPS4R (with GeoTools v3.3.1) running. I have deployed a simple
> annotated R script and successfully tested it with the 52n web client
> (script and request attached). Now I want to make a Java program that sends
> the sensor data, mainly latitude, longitude and water level, as input
> parameters to an R script via WPS request and interpolate them in the
> script, possibly using the gstat library, but I don't know how to do it. So,
> my questions are:

For help in doing the interpolation, I recommend to read the R
documentation and the relevant threads on R-Sig-geo. Also take a look at
the package automap.

> 1. How to request this process? What must be the types of the input and
> output parameters in the XML?

If you want to cerate the request in Java, I suggest to take look at the
Java Client API: https://wiki.52north.org/bin/view/Geoprocessing/ClientAPI

The input type depends on the data you have. In what format do you get
it? Can you read that format in R?

> 2. What must be the types of the input and output parameters in the script?
> How to annotate them?

Please take a look at the demo and test scripts to see usages of the
different types and annotations.

Also, when you have finished your R script, I'd gladly take a look at it
and can try to recommend types for the process.


> I will greatly appreciate if you can give examples because I haven't much
> experience in these technologies.

You can find examples of annotated scripts in the repository:
https://github.com/52North/WPS/tree/dev/52n-wps-webapp/src/main/webapp/R/scripts


Hope this helps,
Daniel

> request.xml
> <http://geoprocessing.forum.52north.org/file/n4026081/request.xml>
> TestScript.R
> <http://geoprocessing.forum.52north.org/file/n4026081/TestScript.R>


--
Daniel Nüst
52°North Initiative for Geospatial Open Source Software GmbH
Martin-Luther-King-Weg 24
48155 Münster, Germany
E-Mail: [hidden email]
Fon: +49-(0)-251–396371-36
Fax: +49-(0)-251–396371-11

http://52north.org/
Twitter: @FiveTwoN

General Managers: Dr. Albert Remke, Dr. Andreas Wytzisk
Local Court Muenster HRB 10849
_______________________________________________
Geoprocessingservices mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/geoprocessingservices
http://geoprocessing.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Raul
Hi Daniel,

Thank you for your response!
The input data is that one returned by the GetObservation in my Java program, but I have parsed it to get just the points (an array of doubles for the latitudes and another one for the longitudes) and the water levels (another array of doubles). These are the data what matters and what I want to pass to R, but I can use them in other formats or even other data from the GetObservation response if it facilitate the work.
I have passed these three arrays to the attached R script (the points are samples; the real ones are more precise and the result have a higher resolution) through the rJava framework (using its REXP type and conversions) and it worked fine. I have used the script version with data frames, although I can use the version with spatial points or other formats if it is easier (see script). With rJava, I have transformed the resulted data frame back to three independent arrays (latitudes, longitudes and water levels) with all the predicted points and stored them in a PostGIS.
Now, I want, basically, do the same with WPS4R, mapping the annotations and types in the XML request (maybe using wps:ComplexData and WaterML?). These data types I have used were the ones I thought it was easier but, again, if there are other types more appropriate I can change. The issue here is that I will offer this Java service (interpolation) as my own service, which must be interoperable, i.e., the input and output data must be manageable by any client without any specific library or offered in a standard way (like via WFS, WMS, etc.), but while inside the Java program I can do specific conversions.

Best regards,
Raul

interpolScript.r
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Daniel
Hi Raul!

Am 28/05/2015 um 04:21 schrieb Raul:

> The input data is that one returned by the GetObservation in my Java
> program, but I have parsed it to get just the points (an array of doubles
> for the latitudes and another one for the longitudes) and the water levels
> (another array of doubles). These are the data what matters and what I want
> to pass to R, but I can use them in other formats or even other data from
> the GetObservation response if it facilitate the work.
> I have passed these three arrays to the attached R script (the points are
> samples; the real ones are more precise and the result have a higher
> resolution) through the rJava framework (using its REXP type and
> conversions) and it worked fine. I have used the script version with data
> frames, although I can use the version with spatial points or other formats
> if it is easier (see script). With rJava, I have transformed the resulted
> data frame back to three independent arrays (latitudes, longitudes and water
> levels) with all the predicted points and stored them in a PostGIS.

> Now, I want, basically, do the same with WPS4R, mapping the annotations and
> types in the XML request (maybe using wps:ComplexData and WaterML?). These
> data types I have used were the ones I thought it was easier but, again, if
> there are other types more appropriate I can change. The issue here is that
> I will offer this Java service (interpolation) as my own service, which must
> be interoperable, i.e., the input and output data must be manageable by any
> client without any specific library or offered in a standard way (like via
> WFS, WMS, etc.), but while inside the Java program I can do specific
> conversions.

If you already have a Java programme wrapping the R script and doing
some necessary conversions, have you considered publishing your analysis
based on that Java code? You can of course call R yourself vom Java
without using WPS4R. Some information:
https://wiki.52north.org/bin/view/Geoprocessing/ExtendAbstractAnnotatedAlgorithm

Can you please clarify what the inputs and the outputs of the process
are? Will the output be empty, but the result is stored in a PostGIS
database?

Looking at your script, it seems like you have vector data (locations
with values) as input. You can encode these in GML, KML, shapefile, ...
or as CSV/plain text. You can also read all of these formats in R, so
they are all viable candidates.

The output is probably a raster dataset, since you do interpolation. So
you could use GeoTIFF as output format.

It is hard for me to say what the "appropriate" type is, that really
depends on what you want to offer to the user. Do you know what kind of
client appications will use the process?


Best regards,
Daniel


>
> Best regards,
> Raul
>
> interpolScript.r
> <http://geoprocessing.forum.52north.org/file/n4026084/interpolScript.r>
>
>
>
> --
> View this message in context: http://geoprocessing.forum.52north.org/Passing-Water-Data-to-R-Script-via-WPS4R-tp4026081p4026084.html
> Sent from the 52° North - Geoprocessing Community Forum mailing list archive at Nabble.com.
> _______________________________________________
> Geoprocessingservices mailing list
> [hidden email]
> http://list.52north.org/mailman/listinfo/geoprocessingservices
> http://geoprocessing.forum.52north.org
> Please respect our mailing list guidelines:
> http://52north.org/resources/mailing-lists-and-forums/guidelines
>


--
Daniel Nüst
52°North Initiative for Geospatial Open Source Software GmbH
Martin-Luther-King-Weg 24
48155 Münster, Germany
E-Mail: [hidden email]
Fon: +49-(0)-251–396371-36
Fax: +49-(0)-251–396371-11

http://52north.org/
Twitter: @FiveTwoN

General Managers: Dr. Albert Remke, Dr. Andreas Wytzisk
Local Court Muenster HRB 10849
_______________________________________________
Geoprocessingservices mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/geoprocessingservices
http://geoprocessing.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Raul
Hi Daniel!

Thanks for your response, I was working in the project with your suggestions.
Now I have the Java and R parts working, but I want to standardize the inputs and outputs (they are all strings).

The inputs are exactly locations with values, and I want to structure them in GML as simple as possible. What I did is attached, but I don't know if it is good or even right (and I didn't want a time element, but it is mandatory...).
The outputs are also locations with points, but a lot of them. I think the best format to it is a raster dataset, but I'm using my own version of R that have not support to raster types and I need to finish this work as soon as possible. So, I'm also using the same GML because it is simple and I know how to store that data into PostGIS, although it produces a quantity of data very large.
What do you think about this format? What else could I use or how to improve this one?

Best regards,
Raul

gml_structure.xml
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Daniel
Hi Raul!

I was travelling last week, my apologies for the delay.

Am 09/06/2015 um 04:19 schrieb Raul:
> Thanks for your response, I was working in the project with your
> suggestions.
> Now I have the Java and R parts working, but I want to standardize the
> inputs and outputs (they are all strings).

> The inputs are exactly locations with values, and I want to structure them
> in GML as simple as possible. What I did is attached, but I don't know if it
> is good or even right (and I didn't want a time element, but it is
> mandatory...).

I am no GML expert, but the attachment you gave is actually O&M, not GML.

Anybody else on the list care to pinch in?


Do you need the input to be well specified, or could you also work with
comma seperated values? Maybe a simpler XML-based encoding, e.g. using
GPX (https://en.wikipedia.org/wiki/GPS_Exchange_Format) would suit your
needs if you have tracking data...

To judge this, please provide a short, plain language description of
your use case. So far we dove in directly to code and encoding, but I
fail to understand what you want to _do_ with the process. E.g. "I want
a process that interpolates about 100 air quality values of a moving
sensor in a small regional area. The client is a web server implemented
in Python talking to the 52N WPS and the process is executed regularly
every midnight."

I apologize in advance if I could not find this in your previous posts
and a pointer to that would be highly appraciated.

> The outputs are also locations with points, but a lot of them. I think the
> best format to it is a raster dataset, but I'm using my own version of R
> that have not support to raster types and I need to finish this work as soon
> as possible. So, I'm also using the same GML because it is simple and I know
> how to store that data into PostGIS, although it produces a quantity of data
> very large.
> What do you think about this format? What else could I use or how to improve
> this one?

GML is not suitable for raster data afaics - have you considered GeoTIFF?


Best regards,
Daniel


> gml_structure.xml
> <http://geoprocessing.forum.52north.org/file/n4026086/gml_structure.xml>
>
>
>
> --
> View this message in context: http://geoprocessing.forum.52north.org/Passing-Water-Data-to-R-Script-via-WPS4R-tp4026081p4026086.html
> Sent from the 52° North - Geoprocessing Community Forum mailing list archive at Nabble.com.
> _______________________________________________
> Geoprocessingservices mailing list
> [hidden email]
> http://list.52north.org/mailman/listinfo/geoprocessingservices
> http://geoprocessing.forum.52north.org
> Please respect our mailing list guidelines:
> http://52north.org/resources/mailing-lists-and-forums/guidelines
>


--
Daniel Nüst
52°North Initiative for Geospatial Open Source Software GmbH
Martin-Luther-King-Weg 24
48155 Münster, Germany
E-Mail: [hidden email]
Fon: +49-(0)-251–396371-36
Fax: +49-(0)-251–396371-11

http://52north.org/
Twitter: @FiveTwoN

General Managers: Dr. Albert Remke, Dr. Andreas Wytzisk
Local Court Muenster HRB 10849
_______________________________________________
Geoprocessingservices mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/geoprocessingservices
http://geoprocessing.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Raul
Hi Daniel,

I want to interpolate a variable number of water level observations produced by static sensors. It is made by a web service implemented in Java and R. The client is implemented in Java but can potentially be in any language, and receives the interpolation result.
I got the most of the work, but the problem is inside the service. It chains several services that exchange water observations (points with values, and some others like observed property) formatted in Strings. Currently I have encoded these observations in a personal format similar to a csv. Now, I want to change this format to a OGC standard because all the rest follows OGC standards, but it can be anyone, GML, O&M, etc..., what be more suitable. This is so that everyone know how to send inputs and parse outputs.

Best regards,
Raul
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Daniel
Hi Raul!

Am 24/06/2015 um 21:27 schrieb Raul:
> I want to interpolate a variable number of water level observations produced
> by static sensors. It is made by a web service implemented in Java and R.
> The client is implemented in Java but can potentially be in any language,
> and receives the interpolation result.

Thanks for the big picture, that helps.

> I got the most of the work, but the problem is inside the service. It chains
> several services that exchange water observations (points with values, and
> some others like observed property) formatted in Strings. Currently I have
> encoded these observations in a personal format similar to a csv. Now, I
> want to change this format to a OGC standard because all the rest follows
> OGC standards, but it can be anyone, GML, O&M, etc..., what be more
> suitable. This is so that everyone know how to send inputs and parse
> outputs.

I would say if you have in-situ measurements that you can model as CSV,
then defining your data in a minimal O&M profile is your best bet. You
could take a look at SWEDataArray [0] for starters, because that is a
bit like "CSV plus metadata".

If you want to create / parse a SWEDataArray in Java, then the 52°North
SOS [1] and common-xml [2] projects might help.

/Daniel

[0] http://schemas.opengis.net/sweCommon/2.0/block_components.xsd
[1] https://github.com/52North/SOS
[2] https://github.com/52North/common-xml

--
Daniel Nüst
52°North Initiative for Geospatial Open Source Software GmbH
Martin-Luther-King-Weg 24
48155 Münster, Germany
E-Mail: [hidden email]
Fon: +49-(0)-251–396371-36
Fax: +49-(0)-251–396371-11

http://52north.org/
Twitter: @FiveTwoN

General Managers: Dr. Albert Remke, Dr. Andreas Wytzisk
Local Court Muenster HRB 10849
_______________________________________________
Geoprocessingservices mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/geoprocessingservices
http://geoprocessing.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Passing Water Data to R Script via WPS4R

Raul
Hi Daniel!

Thank you for these information. They seems to be suitable and I'm working on it. When I get to solve my problem I post here.

Best regards,
Raul