Thread: miniSEED: progress on a new FDSN standard

a) Much of the variability of the 5 proposals are, imho, the result
of different assumptions on the scope of the development: Should we
develop for

- FDSN only (mostly measured data from +/- long living
stations, organized in defined networks, with a community organized well
enough to be able to rely, to some extent, also on conventions)

- Seismology in general (including any type of waveform
modelling, and any type of motion sensors, also poorly identified or
very mobile ones) This requires additional flexibility for identifying
streams from instruments not organized in networks or

- Regular environmental time series in general (leveraging the
possibility of software and data exchange with other domains). This
requires a better separation of time series and domain metadata and
probably an extension of temporal scopes)

ð In order to focus further discussion, I’d like to invite WG2 to take
a decision on this.



b) The question of (backward) compatibility
Miniseed2 started from a subset of seed, with seed covering all
waveforms, data quality metadata, instrumentation metadata, and event
information. In the recent years, separate formats and services for
these topics have developed and were adopted as standards, as it proved
to be more lightweight and flexible to handle different types of
information separately: mseed for waveforms, QuakeML BED for event
information, StationXML/InventoryXML for instrumentation metadata, and
Mustang/WFcatalog with their respective delivery formats for data
quality metadata. So when calling for compatibility of mseed3, we should
more aim at the semantically correct distribution of information into
these four domains than to maintain decisions inherited from full seed.
Otherwise we risk overlaps/collision or gaps with regards to the other
domain formats and respective services

ð I’d like to invite WG2 to aim for a future proof affiliation of
information to these different domains and the respective formats &
services, even at the price of breaking formal compatibility with
seed/mseed2.

c) Application scope and requirements list
There seems to be an implicit agreement that the new format should cover
different application cases (near-real time transfer, economic storage,
efficient seek and retrieval), and that the format should accommodate
for this with different representation variations (including variable
block size, compression, and variable alignment [storage or time]).
However, we define only a file format but none of I transfer protocol,
II subformat conversion rules, or III data access strategy.

ð In order to avoid ambiguities or shortcoming in these different
applications, I’d like to propose 3 add-ons to the requirements list:
-- forward writeability: You should be able to start writing or
transferring a record before having acquired or read all data intended
to be added to it. This detaches the transfer protocol from the block
structure (I) and reduces the needs for subformat conversion. It
disables for data integrity (e.g. crc) in block headers, as those are
available only with all data available.
-- content invariance with subformat conversion. This allows to
losslessly recode the same information in different subformats. It
disallows for features IMplicitly referring to the edges, or extent, of
a data block (as this would change such implicit information)
-- explicit alignment strategy. This allows the user to see whether
block size is aligned to storage units or time intervals, and whether or
not it is subject of change between blocks of the same data source &
stream. It requires subformat indicators, and potentially stream ID aliases



Kind regards,



Philipp









*From:*fdsn-wg2-data-bounce<at>lists.fdsn.org
[fdsn-wg2-data-bounce<at>lists.fdsn.org] *On Behalf Of *Reinoud Sleeman
*Sent:* Mittwoch, 26. Juli 2017 23:23
*To:* FDSN Working Group II
*Subject:* [fdsn-wg2-data] miniSEED: progress on a new FDSN standard



Dear Goran and members of WG2,



the outcome of the successful FDSN WG2 meeting in the Netherlands,
earlier this year, on the future of mini-SEED is

a White Paper which describes the ongoing discussion within WG2 on the
motivation and vision for a new format, the

identification of current needs for a new format, a description of
several proposals and a proposal for the way forward.



This White Paper, which is attached to this e-mail, will be the basis
for further discussion during the FDSN WG2 meeting

in Kobe. At this moment, however, there is no consensus yet about the
adoption and implementation of a new FDSN

standard. Feedback from several stakeholders show very strong, different
opinions which reflect that the FDSN community

is not ready for a new format at this stage.



In order to get broader support to work towards a new format, the FDSN
should first reflect on and take decisions on the

following question:



- What is the scope of FDSN? Does the FDSN envision/support the need to
prepare our infrastructure for large N experiments,

and other more complex experiments, often with lower quality sensors,
that may go beyond seismic data ?



- Can we solve the identification limitation without major overhaul ?

Introducing an disruptive change of the format has an impact on all
stakeholders. Do we accept version incompatibility ?



Because of the different opinions, that we need to respect, and the
potential big impact on the current seismological infrastructure

a careful process must be followed. We only can move forward when the
motivation for a new format is well understood, accepted and

supported by the FDSN. The white paper proposes a path forward, starting
with a requirements list (to be reviewed and approved by WG2)

and the setup of a Technical Working Group to implement draft
realisations of the different proposals with the aim to identify

problems in an early stage and evaluate performance. If the FDSN
supports the design of a new format it will have to be a long

term sustainable format and this takes time to design it carefully, and
even can be more ambitious than the current proposals.

Most likely, SEED was not build in a short time either.



Finally, as soon as the FDSN supports the motivation for a new format it
is required to define a WG2 agreed requirements

list for the new format. This must be the starting point for the
evaluation of the different proposals. That requirements list

will take away current differences in views and opinions of what the new
format should be designed for (e.g. supporting

real-timeor not).



Please find attached the White Paper: The Future of mini-SEED.



Best regards,

Reinoud







_______________________________________

From: fdsn-wg2-data-bounce<at>lists.fdsn.org
<fdsn-wg2-data-bounce<at>lists.fdsn.org>
[fdsn-wg2-data-bounce<at>lists.fdsn.org] on behalf of Goran Ekstrom
[ekstrom<at>ldeo.columbia.edu]

Sent: 21 July 2017 19:54

To: FDSN Working Group II

Subject: [fdsn-wg2-data] miniSEED: progress on a new FDSN standard



Dear Colleagues and Members of Working Group II:



About a year ago (August 6, 2016) I wrote to you concerning the
recognized current need for an updated miniSEED format, and in support
of the ongoing efforts in WGII to develop a new FDSN standard. While I
did not attend the Utrecht workshop on this topic earlier this year, I
understand that it was productive and that much progress has been made
on the detailed definition of a new standard.



I now write again to request that you use the Kobe meeting to take the
next steps toward adoption and implementation of a new FDSN standard.



My understanding is that a nearly complete, though not final,
specification for a new format - miniSEED3 - exists and has been
discussed by the team of technical developers identified at the meeting
in Utrecht. Input came from a number of groups that commented on the
initial strawman proposed by IRIS, and modifications were made based on
input from the Utrecht meeting and through further discussions in the
technical evaluation team.



I will not delve into, or argue for, specific features of the proposed
format. Most of you are more familiar with its details than I. I will,
however, highlight some important aspects of the new format, as
presented by its developers and proponents:



1. It solves the imminent data-center need to identify and accommodate
larger numbers of networks, stations, and sensors.

[Some modern data sets relevant for the FDSN community cannot be
accommodated with the current miniSEED format. ]



2. Conversion of miniSEED2 to miniSEED3, for archiving or distribution,
is simple and lossless.



3. Networks and data centers that do not need the new features offered
by miniSEED3 may not see any benefit in adopting miniSEED3 and can
continue relying on miniSEED2 and their current infrastructure in their
operation.



4. By design, miniSEED3 is not intended to be a low-latency real-time
data format. It primarily addresses issues in data archiving, data
management and data exchange.



Point (1) addresses the critical issue that some data centers have to
deal with now. Having an FDSN standard will strongly encourage data
centers to implement the same standard, facilitating continued sharing
of software and development.



Points (2-3) make it clear that the new format does not impose a burden
on those data centers and networks that do not need miniSEED3.



Point (4) clarifies the limitations of the scope of miniSEED3 and
identifies the separate need for an effort to clarify/develop standards
in the field recording and transmission of data and to address related
issues, such as data latency.



I urge you to use the WGII meeting in Kobe to review and consider the

miniSEED3 format proposal. While doing so, I hope you will not forget
the strategic benefits of having an FDSN standard and also to consider
the risks associated with not adopting a standard. I also hope that your
deliberations will lead to a recommendation that the FDSN Steering
Committee can consider during the second plenary meeting.



My assessment is that the FDSN is ready to, and needs to, move forward
with a solution.



Best regards,

Göran





----------------------

FDSN Working Group II
(http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)



Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)

Update subscription preferences at http://www.fdsn.org/account/profile/






----------------------
FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/