Data Sharing Solution

The City of Philadelphia’s Office of Innovation & Technology is looking for an enterprise data sharing solution to help us (a) publish datasets to the public and (b) share datasets internally between departments. We recognize this may require a stack of products rather than a single product, and are looking for proposals to what that could look like.

Specifically, the solution should:

  • Keep the data up-to-date automatically. We can setup the ETL workflow initially, but there should not be a need to manually intervene to keep the data up-to-date beyond that point (other than troubleshooting).

  • Provide ability to set strict, fine-grained permissions that are difficult to violate unintentionally (the platform will contain nonpublic, sensitive data)

  • Provide an API for each dataset that can handle a good amount of traffic

  • Provide multiple, useful download formats for each dataset (ie. CSV, GeoJSON, SHP)

  • Provide easily built, embeddable visualizations (charts, maps, and tables)

  • Provide a dashboard-building tool for the datasets to departments

  • Provide easy-to-use business intelligence/analysis tools to departments that can connect to the datasets

  • Provide ability for seasoned analysts to use more advanced software to analyze the datasets (this may involve joins and other functionality typically performed at the database level)

  • Provide a DCAT (or DCAT-like) feed of the download links, visualization links, API documentation, etc. to be harvested by OpenDataPhilly.org (CKAN)

  • Store metadata fields that we define

  • Provide a catalog interface for internal/department users to browse available datasets (external/public users do not need a catalog as they will use OpenDataPhilly.org)

Additional Questions

Please post any questions as a comment below the solicitation on bigideasphl.com so that additional information provided in Q&A will be available equally to all vendors. Any questions sent via email will be directed back to the comment section.

Responding

Please send proposals to Tim Wisniewski (tim.wisniewski@phila.gov) no later than Friday, 11/28/2014 at 5:00 PM EST. Submissions should outline how the proposed solution addresses the features listed above and should not propose a cost above $32,000.

 

Posted in Data Management
18 comments on “Data Sharing Solution
  1. Derek Eder says:

    This sounds like a great and ambitious project! Unfortunately, its not a lot of money for a pretty full feature list.

    Considering Open Data Philly (http://www.opendataphilly.org) already exists and is open source (https://github.com/azavea/Open-Data-Catalog/), why not position the RFP as an opportunity to take over and build off that?

    • twisniewski says:

      Hi Derek,
      Great question, and I appreciate the opportunity to clarify a few things:

      There is a separate project underway to migrate OpenDataPhilly to CKAN. OpenDataPhilly will continue to be community-managed and will be the place that the City of Philadelphia directs the public to for its open data. We are looking for a layer behind that to *house* our data, which OpenDataPhilly (CKAN) dataset entries will point to. This could even be another CKAN instance, for example.

      As to the ambition relative to the suspected budget, we’re looking for guidance here. It may be that proposals detail a stack rather than one solution – for instance, an ETL tool, a data warehouse, an API generator, a BI tool, etc. – and we could purchase pieces we don’t have and leverage tools we already do have for some of the rest (for instance, we have a few licenses for Safe FME).

      Our concern with a more traditional proposal targeting a very specific product within a stack that we assume will work is that there may be newer, alternative solutions to these common problems that we haven’t thought of.

  2. Dan Feder says:

    Hi Tim, you write that “submissions should not propose a cost above $32,000. Does this include hosting and ongoing support of the solution after the initial launch? If so, for how long?

  3. Dan Feder says:

    Regarding ETL functionality, do you mean that you essentially need the solution to expose write/update functions via an API, and that any ETL and automation can be done externally?

  4. CSCofNJ says:

    Hi Tim – In processing your response to Derek’s inquiry, could you please clarify the intended use (and expected deliverables) of the $32K max budget? Prior to Derek’s question, the posting implied “product + services”, however, after reading your response, we then felt that the cost for the product would be separate from that budget (thereby, the $32K max budget would be for services). We have a wide array of solutions we could proposed, with ranging complexities and costs. Thank you.

    • twisniewski says:

      Hello, I’m not sure how to clarify it much better. We’re looking for a “stack.” Components of that stack may have costs associated with them, whether “product” or “services” related. Other components of the stack we may already own, such as the Safe FME ETL tool.

      • CSCofNJ says:

        Let me ask a different way…if we recommend a solution/stack of which part of the stack you do NOT own, is the cost to acquire that portion part of the total $32K budget.

        • CSCofNJ says:

          Additionally, is the City open to a full-featured (and financially feasible) full stack replacement? Where existing stack components (like SAFE FME ETL) would be replaced? Please advise.

          • twisniewski says:

            Yes we are open to such proposals. It’s hard to get specific about how the funding would work because different parts of the potential stack may fall into different funding categories (ie. off-the-shelf software like Photoshop vs a subscription like a SaaS product). Typically solicitations like these don’t disclose a budget range at all so hopefully what we’ve provided so far can be helpful.

  5. ewans says:

    Are there any submission requirements beyond and emailed copy, presumably a pdf?

    Thanks in advance,

  6. Aaron Couch says:

    Is the City interested in hosting a solution internally? What kind of hosting could you support? If the city is NOT able to provide its own hosting infrastructure, does the $32K budget need to include hosting, and if so, for how long?

    • twisniewski says:

      Hi Aaron, we would prefer a cloud solution if possible – either externally hosted or as a SaaS product – rather than hosting the solution on our infrastructure. Ideally proposals would include at least a year of hosting (or service in SaaS terms) at that budget.

  7. Dan Feder says:

    Tim – are you hoping to integrate this with the city’s existing API key management system? If so, should that be part of the proposal or would that be a separate contract?

  8. CSCofNJ says:

    Tim – Do you have an anticipated date or even a timeframe as to when you will be making a decision (notification) from the responses you’ve received? Would appreciate the insight. Thank you.

Leave a Reply