Data Sharing Between State, Local, and Federal Agencies



Podcast 5

Data Sharing Between State, Local, and Federal Agencies

Topic/Title: "Data Sharing Between State, Local, and Federal Agencies"

Description: The State of New York received a COPS technology grant to develop a statewide criminal incident database based on the National Information Exchange Model (NIEM) standards and integrated with federal, local, and other state systems. It will ensure that incident information meets state fusion center and FBI N-DEx (Law Enforcement National Data Exchange) program needs.

Participants:

Christine Tyler

Project Director

New York State Division of Criminal Justice Services

Ben Krauss

Public Safety Technology Specialist

SEARCH, The National Consortium for Justice Information and Statistics

Recorded: April 13, 2010

Podcast Length: 26:06:00

Ben Krauss:  The following is another in a series of recorded audio interviews distributed via websites and podcasts on lessons learned and best practices from projects funded through the COPS Technology Grants. These podcasts are presented by SEARCH, The National Consortium for Justice Information and Statistics, through funding through the U.S. Department of Justice Office of Community Oriented Policing Services, COPS, Cooperative Agreement 2007-CK-WX-K002.

Today's topic is data sharing between state, local, and federal agencies. Our guest today is Chris Tyler, project director for the New York Division of Criminal Justice Services. I'm Ben Krauss, a public-safety technology specialist for SEARCH and moderator for this podcast.

The state of New York received a COPS technology grant to develop a statewide criminal-incident database based on National Incident Information Exchange Model, or NIEM, standards, and integrated with federal, local, and other state systems. It will ensure that information needs meet state fusion center and FBI N-DEx program needs.

Chris, we'd like to welcome you here today. Can you tell us a bit about NYDEX and share your background with it?

Chris Tyler:  NYDEX is a New York State data exchange project for local law enforcement agencies across all of the 66 counties of New York State. We hoped to bring together a statewide data repository and give access to that repository to local law enforcement agencies. The men on the street, essentially, are who this is intended for. So it would assist them in the criminal investigation process. And we've been at it for a little less than 18 months. My background, essentially, is I've been in the public safety, criminal justice space since about 2003. I worked for IBM prior to New York State, as a consultant. And in that capacity, I worked with the New York City Police Department in the construction and deployment of the Real Time Crime Center. So I have some background in bringing together crime analysis information and then providing access to that information to local law enforcement officers on the street.

Ben:  Excellent. Thank you. Through NYDEX, New York criminal justice agencies will have an automated service oriented means of linking records statewide and beyond. How will agencies contributing incident-based data benefit from this local, state, and federal partnership?

Chris:  The system's intended to be sort of a give-to-get structure. So the more information a local law enforcement agency is willing to share, the more information they'll have access to as well. The long-term goal is to provide public safety, for both the officer on the street and the public at large, during the investigation of a crime. Speed to resolution of the case is always a priority for police agencies. They say, statistically, that if they can solve the crime within the first 48 to 72 hours, their success rate is much better. Efficiency in the resolution process is important to them, so they want to get as much data as possible in their hands that provides them information into people, places, and activities surrounding the crime activity.

And long term, they'd like to be able to be proactive in prevention of crime, so they're hoping to be able to provide pattern and trend visibility to criminal activity surrounding events like a parade or a holiday or a national event like a political campaign or that type of thing. So, that's the long-term goal, but the basic goal is to just solve crimes and give the policemen on the street the ability to do their job better, faster, more efficient.

Ben:  Excellent points and a very clear purpose. Thank you. How'd you get buy-in of participating agencies?

Chris:  Well, actually, we have executive sponsors from three of the largest organizations in New York State. The New York State Police is an executive sponsor, the New York State Chief of Police Association has also sponsored, and the New York State Sheriffs' Association. In DCJS [Division of Criminal Justice Services], we have a public safety officer, who is also a sponsor. And long term, our goal is to get other criminal justice agencies, like Homeland Security, Corrections, Probation and Parole, in this repository as well to share their information. So, we've been pretty lucky that, statewide, data sharing is an initiative at even the governor's levels.

Ben:  Excellent. Thank you. You said that the DCJS collects incident-based data for agencies that use upwards of 20 different records management systems [RMS]. Tell us a bit about the RMS that your agency provides.

Chris:  DCJS created a records management system. It's called the Spectrum Justice System [SJS]. It is deployed today to 240, maybe a little over 240, of the 520-plus agencies that exist in New York State today. The reason we created it was really just to provide an alternative, cost-effective, records management system to serve the vendor applications that are out there that might be cost-prohibitive, especially to the smaller, local law enforcement agencies who either don't have the IT staff themselves or have the funding to enable a third-party vendor system on their behalf. So, that's how it came about in New York State. Clearly, the majority of agencies have taken advantage of it. We continue to support it and deploy changes to it as they need to occur.

Ben:  Excellent. Thank you. Could you please confirm how many agencies use it?

Chris:  240. Today. We're actually going to the next version deployment. So by the end of the year, we'll expect to have version seven deployed to all of the 240 agencies.

Ben:  Thank you. And could you please confirm why the state provides it, the primary reason?

Chris:  The primary reason was just to provide a cost-effective structure. Funding's limited in New York State, especially to smaller agencies. We give it to them at no cost. So if they have limited funding or the records management system that they might be attempting to buy would require modification to get to what they need for customization that could be cost-prohibitive for them.

Ben:  Thank you. Where are you at in the process of upgrading the RMS for NYDEX?

Chris:  What we've done with the RMS is add an extract, transform, and load [ETL]process. ETL is the acronym often referred to from a data-warehousing perspective. So we built an ETL process as an adjunct to this system. So we didn't actually update the application itself; we created an extraction process that is deployed with the application on the local server. We built the structure, and several components are a part of that structure, and we're in the process of integration-testing that process now. So it's built, we're testing to make sure that we've gotten it right, and we'll be testing the exchanges with three or four of our locals within the next couple of weeks.

Ben:  Thank you. How do you plan to help agencies using other records management systems shared through NYDEX?

Chris:  Well, one of the goals in building the process for the SJS, which is our system, was to build it on components that could be reusable by other organizations. So we built the ETL process in an open platform, open systems platform, called Kettle. It's a Kettle extraction program similar to many of the ETL cost products that are out there in the major player space. So we built it in that process. We built the rules engine, which is a component of the deliverable to the locals as well so they could set security structures against their data. We built that in Java so that it's reusable to anybody who wants to pick it up and use it.

And so from the point of extraction from the database forward, once the data is acquired by the ETL process all of the transformations and components necessary to transmit the data to DCJS are included in this package of components, if you will. So the work on the part of anybody who's a third party vendor or a custom application will be to map their data to the ETL extraction process. And then the rest of it is reusable.

We wanted to try to leverage as much of the work we had done for other people and try to reduce the cost to the local agency at the end of the process. So that's why we did that.

Ben:  Thank you. It sounds from our previous discussions as though working with the FBI has been a more positive part of this project thus far. How has the concept of operations document helped advance the project?

Chris:  The FBI has done a lot of work since I think early 2003 in the development of this system that largely mimics work we're trying to do at the New York State level. So their concept of operations document was essentially the business requirements document that they created as a result of their numerous conversations with the end users that they interviewed in order to define what needed to be in a system to make it successful. And they modified that document as they've gone along from iteration one to two and beyond. So they continually update that for changes to the system as a result of use of the system which actually has been in process. Iteration one was deployed in I think 2005, maybe 2006. So they've had some experience with live customer use.

And that feedback loop is always beneficial so they gave us access to that document. They come to the table to help us with data mapping structure completely. Anything we needed to get from them they've been more than willing to assist us – training end users on their system, enabling seamless access to their system through our portal, all of those types of things so that we didn't have to duplicate the effort that they created.

We've been able to leverage and they've been a partner through the entire process so it's been very beneficial, very helpful, a very pleasant experience actually.

Ben:  Thank you. Are there any other notes referencing their participation that have been helpful? Any other elements?

Chris:  They're assisting us in marketing the program. So N-DEx does essentially what NYDEX wants to do at the national level. And the goal for New York State is to transmit our data to the federal level so that we're not only sharing within our borders but external to our borders with our neighboring states. The closest of course would be New Jersey, Connecticut, Rhode Island, the upper northeast, Massachusetts. New Jersey is currently a participant of N-DEx, and then it's more like the I-95 corridor, people who are current participants closer to the DC area. But we envision that all of our neighbors will be in it as well. So marketing this across the state in concert with our own marketing efforts has been very good. Giving us components and technology that they've already built for reuse for connecting to their database, there's a search component that they provide so that we can federate to their database.

The IEPD [Information Exchange Package Documentation], which is the document that creates the data standards that you have to live by to share data with the feds, is a published document that they give you. And they bring their resources to your organization if you request it to help you do the mapping. And they've done all that for us.

Ben:  Excellent, thank you. How do you foresee agencies using FBI analytic services?

Chris:  Well, some of the services they have are link analysis tools, and you'd have to really know the criminal law enforcement space to know what that means, I guess. But for those out there who are hearing this message who do understand link analysis, it allows you to take a person like Chris Tyler and form all of the ties to that individual that might exist in a database structure as you start to assimilate more and more data. So everybody I know, every place I've lived, every phone number I may have had, every relationship that has some relevance to the law enforcement community might be available to do that link analysis. And that's a powerful tool. So the FBI has already built that tool. I don't have to rebuild it. I can just direct them to that tool and they get the benefit of both national and statewide data to perform that type of analysis.

Early in the process I talked about patterning and trending so that you could prevent crime rather than just responding to it. And you need those types of technology in order to get to that place of proactive policing and anticipating the crime before it actually occurs. So that's what I see it as and all of the law enforcement end users who have seen this system are very impressed with what they've seen.

Ben:  Thank you. What challenges have you faced with data mapping, extraction, transformation and loading?

Chris:  Well data mapping in any data warehouse project is always the most intense of the processes to be done. There's a lot of making sure the element that you started with maps to the right place and that the definition of what that data is is presented correctly. The standards for federal data sharing are pretty stringent. So you have to follow a strict set of rules on how to present that data in a sharing method, and it's a little more complex than just normal data extraction and transmission. So the level of effort in the data mapping was intensive at best and oppressive at worst, I guess is what you'd say. It's a challenge but it's not undoable. But it's a challenge. It's always a challenge.

Ben:  Thank you. Have there been challenges finding people that are skilled with the needed technical skills?

Chris:  There have been challenges in finding resources who understand the NIEM construct. We did have some help from SEARCH early on in the process to gain an understanding of the NIEM conventions and what that means from a transformation perspective from raw data to a NIEM exchange. So we did have some assistance in that vein. And the FBI has some of that knowledge as well. So we were able to glean understanding from organizations who'd done it better, but even those two organizations have limited resources. So if you have a major initiative, it could be a risk finding resources who understand the process.

We've built those resources in New York as a result of this process. They didn't exist here before this project actually occurred. It is a DCJS standard to try to share information in the justice model standards. So that's a benefit to us that we've gone through this exercise but it was a challenge to get, or create or build those resources.

Ben:  Thank you. How about project management skills?

Chris:  Our organization actually has focused a lot on the PMO [Project Management Office] management structure in the past three to four years, so they had a pretty strong project management office in place. Most of the individuals who are a part of that organization have project management institute credentials, myself included. That's the type of individual you need for a project of this size where it crosses all of the internal boundaries and even some of external organizational boundaries to coordinate the effort. We were successful because we had those in place. An organization that doesn't have those in place should probably think about going down that path. It's a good strategy. It works well.

Ben Krauss:  Excellent. Do you think that the project management credential is very important?

Chris:  I do, I do. I think the PMI Institute does a lot to build their project management organization. I think their testing methodology is effective. Their training is more than efficient.

Ben:  Excellent. Thank you. What value has open source software brought?

Chris:  Well, I talked about open source software earlier when I talked about the ETL process that we built to add on to our RMS system. The real benefit of open source is that it's out there in the community. Many people use it. It's been tested and tried and, last but certainly not least, there's no cost to enabling it to start with. There's a cost total, cost of ownership long-term because you have to continue to maintain whatever you built in that product set. But you get a pretty good out-of-the box foundation to begin with in open source. If it has any presence in the community and if it's been there for any length of time, it's a benefit. Other people can pick it up and reuse it as well. There's none of the licensing and costs associated with purchasing or transferring software between organizations or agencies.

Ben:  Thank you. Are there any downsides that you see to open source software?

Chris:  Some of the downsides can be that it's a higher development cycle timeline. It could have more requirements in the maintenance long-term because you're essentially taking ownership of it once you've developed in it. The one we utilized was similar to some of the standard ETL products out there so, if you had any expertise in any of those other product sets, it would be an easier lift. If you have none of that expertise, it might be a larger lift.

Ben:  Thank you. You went through some grant reprogramming as your project progressed. Why was that necessary?

Chris:  At the beginning of any grant process, you put down what you think you can accomplish. Then your budget is formulated based on what you think you know at that point in time. That changes over the course of a project cycle almost without exception. Small projects, large projects, they all deviate to some degree in your vision of how you will deploy them and how you actually deploy them. For DCGS, we started thinking we would build a statewide system and all of the things that you needed to build to enable that system internally. When we realized the FBI had done a great deal of that work already and that there was opportunity to leverage that work from the perspective of giving more, adding value to our end customers' delivery rather than just duplicating effort, we had to shift gears. We changed our technology mix and we changed our resource mix as a result of the change in direction.

Some of that would have happened anyway without the change of direction because you choose a set of tools you think at the definition of the grant. Then the tools that make more sense for your agency from an enterprise perspective sometimes deviate from what you thought at the time you authored the grant. So, all of those things came into play. We chose different technology. We used more resources. We coordinated our effort with the FBI. Our direction and our technology-resource mix changed.

Ben:  What lessons did you learn?

Chris:  The lesson that we learned was that getting to that redefinition of the grant sooner than later is useful. We're pretty much down the path of delivery here so it's been like 18 months in process already. We have probably over two years in process actually. So there's a limited time left till the end of the grant so we're reprogramming farther down the road than we would have liked to. We got all the approvals we needed but it was a bit of an effort that far into the project to have to provide all of the documentation necessary for the reprogramming structure for the team at the COPS Office. That was a little bit of challenge. The grant reprogramming is time consuming both, I think, at the federal level and the state level. You're putting all the documentation together. They're trying to understand everything that you've put together. There's kind of an iterative back and forth till that understanding meshes. It's kind of time consuming.

Ben:  Excellent, thank you. In closing, what are your takeaway messages for the success of others undertaking similar projects?

Chris:  I guess I would advise them to not underestimate the effort in the data mapping exercise. That tends to be the case in most data sharing initiatives that you underestimate what's really involved in understanding the data and then mapping it to a structure that makes sense for delivery to the end user. Interaction with the customer is essential to get that information right. The way they want it delivered, what they're comfortable with, the format that they're comfortable with seeing it, is important. So a lot of customer interaction is a good thing. We did several roundtable-kind of structures with our end users which proved to be very helpful not just from... Because we had talked earlier about the executive sponsors but the real buy-in is at the local level. The people who are going to use this system have to understand what's coming, how it benefits them, and how it might be useful in the use of their job. That communication and that relationship building is essential to the success of a project like this. It was the same when I did it at NYPD. It hasn't changed in this space either.

Leveraging the work of others, I would commend anyone to look at what the FBI has built if they're trying to get into a statewide/nationwide data share at the incident level and beyond. Incident and arrests are the foundation of any criminal investigation process so any data beyond that just adds breadth and complexion to the investigation process for the user. If you set a firm foundation, which the FBI has done by building their N-DEx system at the incident and arrest level first... That's how the NYPD Real Time Crime Center was also built with the foundation of incident and arrests.

So that's the core for data sharing in the criminal investigation space and everything. If you do that right and you do it well and you leverage what other people have already done, you're halfway there before you've even started the game. I think that would be a bag to anyone trying to... Certainly I would interview or spend time with the people who've gone before you. Because they've already encountered some of the pain associated with going there and they can give you lessons learned and help you avoid some of those pitfalls.

Ben:  You provided some excellent, very helpful information. I want to thank you so much.

You've been listening to a SEARCH podcast on data sharing between state, local, and federal agencies. We'd like to thank our guest, Chris Tyler, Project Director for the New York State Division of Criminal Justice Services.

This project was supported by Cooperative Agreement 2007-CK-WX-K002 with the U.S. Department of Justice Office of Community Oriented Policing Services. Points of view or opinions expressed in this podcast are those of guests and moderators and do not necessarily represent the official position or policies of the U.S. Department of Justice.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download