Thread: FDSN web service specification changes proposed by IRIS

Thanks for the feedback.

1) Regarding ambiguity in the "Required" column of the parameter table. Good point, a sentence can be added after each service-specific table to clarify that Required refers to support and not each request. Alternatively, the column can be called "Support" and the entries can be "Required" and "Optional". Any suggestions or preferences? Earlier in the spec is language strongly imply that this designation refers to support, but it is not very close to the table.

Support, with Required and Optional, would work, or 2 sections in each table to separate required supported parameters from optionally supported parameters.

2) Regarding the text output in CSV. The 'text' output (event or station) can include commas in certain fields, comma-separated values are then mis-parsed. Schemes like quoted fields to contain or escape embedded commas can be a pain to parse, where as vertical bars are easily parsed in shell scripts, etc.

I agree quoted CSV is tricky in shell scripts, although most languages have parsers available that handle this.

It would also be useful for each of the columns in the format to be specified, for example what is the difference between eventid and contributorid?

The general comments:

3) Regarding GET and HTTP caching. There are a number of use cases where a huge number of GET requests becomes a real limitation due to the overhead in the requests (thousands of requests) that the POST request capability resolves. In our experience with services so far there are very few requests that are exactly the same and therefore very limited use of HTTP caching at a level that is unaware of the data characteristics and therefore cannot subset data or recognize when to subset.

Agreed that many users will submit unique requests. However, web applications that consume a web service can make requests consistent across many users, allowing regular caching to be effective and keeping resources available for unique requests.

I'm reluctant to see POST method support _required_ for something that would otherwise still be possible using GET.

Maybe station web services are used differently from event web services.

4) Regarding HTTP status 413 usage. I agree that the HTTP spec states this is only for the request entity, for FDSN service it should technically only be used to communicate the passing of a limit on the size of a POST body that a service interface will accept.

HTTP status 400 does not seem like a good fit either, it specifically means "The request could not be understood by the server due to malformed syntax.", which is not the case. At least a 413 has something to do with size, the message returned with the error can clarify that it refers to the size of the response.

There seems to be no HTTP status for "Response entity is too large". Are there any other suggestions for how to communicate this condition?

After re-reading, I agree that 400 is more appropriate for syntax errors.

Maybe 403 Forbidden: "the server understood the request, but is refusing to fulfill it." Text describing how it exceeds the server limit would explain why the request is forbidden.

Thanks,

Jeremy



Chad

On Sep 19, 2013, at 2:37 PM, "Fee, Jeremy" <jmfee<at>usgs.gov> wrote:

Hello,

I have a few comments after reading the updated specification:

- The "Required" column in the specification is a little ambiguous. Either 1) the parameter is required to be supported, which I suspect is the intent, or 2) the parameter is required for each request.

- The event "text" format looks a lot like CSV, why not CSV?


And a couple general comments, not specific to the latest changes:

- The POST method circumvents any HTTP caching. I would suggest this be optional, because the requests could be broken into smaller GET requests that are cacheable.

- HTTP status 413 (Request Entity Too Large) refers to the _request body_ being too large, rather than the _response content_ being too large. It's possible an application firewall inserted into a network between the provider and client would generate this response even if the query would otherwise be acceptable by the provider; particularly with POST requests.

One alternative would be 400 Bad Request when exceeding search limit, and specified "limit" and "count" methods so an agent could determine the result size was too large.


Thanks,

Jeremy



On Wed, Sep 18, 2013 at 10:39 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

Hello WG III members,

As presented at the WG meeting in Gothenburg, Sweden, below is a list of FDSN web service additions proposed by the IRIS DMC. With the exception of the POST submission method to fdsnws-station, all of these changes are in production at the IRIS DMC.

To provide further details and clarify how the changes would alter the specification, a draft version is attached with additions highlighted in red.

We proposed that these additions to the 1.0 specification become version 1.1 of the specification.

regards,
Chad


Proposed changes:

* Designate optional versus required parameters for each service interface. Effectively defining a minimum level of functionality that clients can expect. The WADL return from each service interface specifies which of the optional parameters are supported.

* Allow fdsnws-station to accept arbitrary selections via the POST method. Support for this method is required for all 1.1 implementations of the service. The request is to be formatted similarly to the POST selection method supported by fdsnws-dataselect, where parameter key-value pairs are followed by a block of windowed channel requests.

* Add optional 'format' parameter to all three interfaces. The default value for the fdsnws-station and fdsnws-event services is 'xml' (returning FDSN StationXML and QuakeML respectively), an additional value of 'text' is defined as returning results in a simple ASCII format (illustrated in the specification). The default value for fdsnws-dataselect is 'miniseed'. Future expansion of the accepted values for this parameter are anticipated to return the results in alternate formats.

* Add optional 'nodata' parameter to all three interfaces accepting values of '204' and '404' to control the HTTP status code returned when a successful request matched no data. The default value is '204'.

* Add optional 'matchtimeseries' parameter to fdsnws-station to limit the results to matching data availability times.

* Clarify definition of starttime and endtime parameters to explicitly include intersection with metadata epochs.

* Clarify definition of minimum and maximum latitude and longitude to be inclusive by adding "or equal to".

Except the POST method addition to fdsnws-station, all other additions are optional. Service implementations should return a WADL which appropriately advertises the optional features supported.









_______________________________________________
fdsn-wg3-products mailing list
fdsn-wg3-products<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/fdsn-wg3-products

Dear Marcelo,

Thank you for your comments.

1) True, it is only implied that the time criteria are tested against
different values depending on the level. This is not limited to the
text output but also applies to the XML. I have no issue with adding a
description of expected results. By the way, the IRIS DMC's
implementation of the service always applies the time criteria to
channel times to simplify the coding. The assumption is that channels
"fit" within stations that "fit" within network time ranges. The
consequence is that we cannot report networks or stations that do not
have channels, so far this has not been a problem as we do not have any
such metadata.

2) We have to strike the right balance between too many fields in the
text output versus keeping it small and useful. In my opinion it is OK
to leave restricted status out of the text format. A caller can still
limit the response to include restricted stations or not using the
includrestricted parameter, this fulfills the main use case of
restricted status.

3) I have no problem changing the header name in the comment field. It
is informative, but also designed to match the Instrument field of a SAC
header. In the IRIS implementation this actually maps to the
Network::Station::Channel::Sensor::Type field in StationXML which is the
"instrument identifier" field of Blockette 52 in SEED. I propose we
stick with something explicit such as "Sensor" or
"InstrumentDescription" instead of anything generic, otherwise the field
will not be very useful. I also do not think a URL is needed, a user
already knows the URL to get more information, from the same service it
requested the text from.

At IRIS we have used this text format for a couple of years now and
found it to be very useful, taking the place of the XML format for many
needs. Regarding changes to this format I propose we first try this
text format and discover where it is lacking, in the future we can
define other text output formats are needed. There are many use cases
and no one simple set of metadata will solve them all.

best regards & obrigato,
Chad

On Sep 23, 2013, at 12:20 PM, Marcelo Bianchi <m.bianchi<at>iag.usp.br>
wrote:

Dear Chad, All

Thanks for the draft for the 1.1 version of the specification. Even
that I was not at the meeting in Sweden I hope I can contribute with
some comments for the document, more specifically on the ASCII format
definitions proposed. Beyond that I think that in general they are nice
additions to the specifications and my comments follows:

1) The StartTime/EndTime fields changes meaning depending on the value
of the level="value" parameter. We should add a notice for that
behavior. When level=network they are network operation times. When the
"level=station" is in effect station operation times.

2) I miss the restricted flag on the ASCII format. Should we add a
column on all levels to indicate that the network/station/channel is
open/restricted/some like in the XML version?

3) I could not understand what would be the contents for the
"instrument" field before looking into one of the given examples (it is
not straight forward from its name). I would suggest to rename it to
InstrumentDescription (or just description) ? Moreover I don't oppose
myself on having this field there, but I don't see any real usage, do
you have already something in mind or it is just an informative field?
I would still suggest that would be more useful to have a URI/URL where
the user could fetch the response for the channel inside instrument
field than the description itself. But for now, this is just one idea.

With my best regards,

Marcelo Bianchi

Em Mon, 23 Sep 2013 11:19:49 -0300, Clinton John
<jclinton<at>sed.ethz.ch> escreveu:

Hi Chad,

thanks for proposing these changes. We at the SED are happy to see
these adopted in v1.1.

John


On 19 Sep 2013, at 06:39, Chad Trabant wrote:


Hello WG III members,

As presented at the WG meeting in Gothenburg, Sweden, below is a list
of FDSN web service additions proposed by the IRIS DMC. With the
exception of the POST submission method to fdsnws-station, all of
these changes are in production at the IRIS DMC.

To provide further details and clarify how the changes would alter the
specification, a draft version is attached with additions highlighted
in red.

We proposed that these additions to the 1.0 specification become
version 1.1 of the specification.

regards,
Chad


Proposed changes:

* Designate optional versus required parameters for each service
interface. Effectively defining a minimum level of functionality that
clients can expect. The WADL return from each service interface
specifies which of the optional parameters are supported.

* Allow fdsnws-station to accept arbitrary selections via the POST
method. Support for this method is required for all 1.1
implementations of the service. The request is to be formatted
similarly to the POST selection method supported by fdsnws-dataselect,
where parameter key-value pairs are followed by a block of windowed
channel requests.

* Add optional 'format' parameter to all three interfaces. The
default value for the fdsnws-station and fdsnws-event services is
'xml' (returning FDSN StationXML and QuakeML respectively), an
additional value of 'text' is defined as returning results in a simple
ASCII format (illustrated in the specification). The default value
for fdsnws-dataselect is 'miniseed'. Future expansion of the accepted
values for this parameter are anticipated to return the results in
alternate formats.

* Add optional 'nodata' parameter to all three interfaces accepting
values of '204' and '404' to control the HTTP status code returned
when a successful request matched no data. The default value is '204'.

* Add optional 'matchtimeseries' parameter to fdsnws-station to limit
the results to matching data availability times.

* Clarify definition of starttime and endtime parameters to explicitly
include intersection with metadata epochs.

* Clarify definition of minimum and maximum latitude and longitude to
be inclusive by adding "or equal to".

Except the POST method addition to fdsnws-station, all other additions
are optional. Service implementations should return a WADL which
appropriately advertises the optional features supported.

--
Dr. Marcelo Bianchi
Centro de Sismologia da USP
Instituto de Astronomia, Geofísica e Ciências Atmosféricas (IAG/USP)
Rua do matão, 1226 São Paulo/SP 05508-090 Brasil
Phone: +55 11 3091-4743 Cel: +55 15 99134-8532
http://moho.iag.usp.br/ http://www.iag.usp.br/