Smart Grid for a Downtown Network: Unique Challenges for Network Protector Communications

Avista’s first effort to gather data from its network protectors for planning and operations didn’t occur until 2001 after many other utilities had already developed such communication systems. Communicating with the network protectors provides system visibility that is otherwise unavailable.

Yet 20 years later, after many failed attempts, a Downtown Network Vault Integration System is finally becoming a reality.

Avista has served the downtown core of Spokane, Washington, with an underground network distribution system since the early 1900s. The buried network was constructed by Washington Water Power, which envisioned Spokane as a future hub for the Inland Northwest. Encompassing the area from Maple Street to Division, and 3rd Avenue to the Spokane River, the system is far more reliable than radial power lines found throughout the rest of Avista’s system.  By 1930, there were 400 underground structures. Today, there are more than 800.

Avista cablemen Stephen Doan and Marcus Spradlin back a fiber splice trailer into a traffic control zone near a transformer vault.

A timeline of integration efforts

The earliest record of design scoping for a Network Vault Integration System at Avista occurred in a memo dated Sept. 11, 2001. The idea was batted around for several years before an initial shop-based pilot was attempted in 2004–2005.

This was followed at typical utility speed by a small field pilot in 2009, where several network protectors were connected via twisted-pair copper wiring back to the main network office (but not to the centralized SCADA system). However, this led to unusual “operational authorities.” For this reason and others, the project was put on the back burner for another seven years.

Efforts to scope the project restarted in 2015. However, technologies had changed, which made most of the lessons learned from previous trials obsolete. Fortunately, fiber optics had now become the preferred method of communication.

In 2017, an official project charter and scoping memo were written. Construction project management experience was added to the project and capable design consultants helped the process. However, it was the willingness to find dollars in existing budgets that was key. As the design was introduced in the field, cooperative expertise from around the company worked with a shop full of cablemen eager to see the job finished. Today, construction is nearing 50% completion.

A network protector is a type of electric protective device used in networked electricity distribution systems. Its function is to automatically disconnect its associated power transformer from the secondary network when the power starts flowing in the reverse direction, and reconnect when it detects that powerflow returns to normal.

The tangled path of design

Many things were “known” by those closest to the project before design efforts started in earnest, and while some supposed certainties were correct, others were not.

For example, fiber optics were preferred for intervault communications paths. While technologies in the industry allowed for copper wiring in both intervault communications and SCADA backhaul, fiber held two key advantages: electric isolation and bandwidth. However, an initial interest in excluding some departments of the company (IT specifically) from the installation and future maintenance of the project led to the need for fiber to behave similar to copper — i.e., the communications would need to be radial from a central point to all vaults (star topology).

A consultant’s analysis led to the identification of key facts about this design preference:

1. A star topology would require the crews to pull in more total fiber cable length, approximately 30% more footage, compared to the other choices.
2. Star topology, depending on how it was implemented, would require different types of fiber cable for different runs (larger bundles near the point of origin, with smaller bundles as the fiber reached the edges of the network).
3. The number of splices required was double or triple with a star topology, depending on how it was implemented.

Avista’s Network Avista’s downtown network system consists of four 120/208V secondary street grid networks, fed by fourteen 13.2 kV primary feeders out of two urban substations. There are approximately 180 transformer vaults; around 30 of them serve spot networks while the rest supply the street grid. Spots may be 120/208V or 277/480V, while the street grid is 120/208V.

Instead, the consultant recommended that IT be asked to lend support to the project by setting up small and relatively simple Ethernet switches in a looped ring configuration. This design approach still included two hosts or “marshaling” locations — one associated with each substation whose feeders supply the downtown network, and each providing a backhaul route to SCADA. The overall reduction in cost caused by adding managed Ethernet switches was accompanied by an increase in communications reliability.

SCADA Communications Network Diagram

Another important decision was to employ a data concentrator at every single transformer vault. Again, this ran contrary to prior efforts, where fears of overall cost dictated the size of the project. In this case, while available technology will support multiple transformer vaults downstream of a single localized data concentrator, a data concentrator (and associated fiber connection) to each vault is a solid platform that allows for future buildouts of additional connected IEDs at each vault (e.g., transformer monitors, water sensors, ambient sensors, and even camera technology).

Last, it was important to properly identify the operating authorities for this “new” electrical system. For over a century, the only hands to control network protectors belonged to cablemen, which had led to the downtown network system being essentially ignored by those who ran the rest of the electrical system at Avista.

A culture change was needed for the project to succeed. In a world where “201” or SCADA operation of substation breakers and mid-line reclosers is ubiquitous, the network protector was always an anomaly. At Avista, this problem was split into two:

1. Prior integration efforts had ended with a pilot system that was remotely controllable from a (non-union) downtown network engineer’s desk. At Avista, whose workforce consists of union cablemen and operators who are systematically trained to ensure operations are safely coordinated, this made everyone uncomfortable.
2. Avista’s downtown network had always been under official purview, by way of switching authority, of the transmission operations desk. While no remote control was available, switching orders were always routed there and centralized troubleshooting ran through them. At Avista, the rest of the distribution system (35 kV and below) had historically been operated by a separate distribution operations desk. The downtown network had been an outlier — one that went unnoticed primarily because of a complete lack of SCADA.

Avista journeyman Gordon Harvey and Stephan Doan wrangle extra fiber cable length through watertight box grommets inside a transformer vault.

Reconciling both of these points was key to moving forward, as elements of both problems had stalled the project several times over almost 20 years. To that end, special care was taken to note that no remote control by engineering would be possible in this project. Beyond that, discussions were put together with interested stakeholders to discuss, determine and document a likely move of the downtown network from transmission operations to distribution operations’ purview. This move involved switching procedures and SOP creation, training, and even a transition to radio communications in the field. This decision was not made lightly.

Lab installation

Previous attempts to integrate the downtown network protectors started with a pilot phase, followed by testing prior to the main construction rollout. But the furthest any proposals to integrate had progressed to was a pilot installation. In 2018, it was decided to use a more robust design process in lieu of a pilot installation. But there was still a need to “field” test the design, integrating devices from multiple manufacturers.

A lab was created in the downtown network shop’s training room, which incorporated a marshaling rack, a vault communications cabinet, and several network protectors. This provided a more convenient environment to test and troubleshoot end-to-end communications and functionality than in an actual network vault. In addition, it will provide a safe training environment, from both personnel and equipment perspectives, for those working on the integration system.

Construction (the slog)

With a well-thought-out design in hand, and stakeholders bought in, the cablemen were excited to get started. The project had been scoped with a three-to-four-year construction phase, dependent upon budget and resource availability.

The first task was pulling the fiber rings, starting with the largest of Avista’s four networks.

In order to open enough manholes to make the fiber pull, and keep a top man near every hole, the Downtown Network Shop had enlisted assistance from the Spokane Line Dock, which provided around 20 journeymen for the first night “pull.” With a traffic control company set up from I-90 straight into the heart of downtown, the discovery of a missing conduit path for the initial “home run” pulls at 10:30 p.m. was not a good way to start. The crew made the most of the night, however, and skipped down the line to pull in more of the 30+ different fiber runs on their list.

Multi department support from both the Downtown Network shop and the central Avista Lineman Office was required on many of the fiber pull nights due to the extraordinary number of open manholes. Foreman Zach Muller’s pre-job tailboard sessions were key to both safety and success on these night jobs.

It took five overnight efforts, along with daytime support for shorter pulls in less impactful areas, in order to complete the fiber ring installation in the Metro West Network. Several months later, the “home runs” of the fiber were finally installed.

As the fiber was pulled in, more daytime work was put in by a single downtown network crew paired with a communications shop fiber splicer. Extra feet of fiber had been pulled into each transformer vault in order to allow the fiber to be brought up out of the vault entrance and into a clean fiber splice trailer. After the fiber had been terminated by the splicer in the trailer, and tested, the cable had to be wrangled back down into the vault and be coiled/racked on the vault wall near the communications cabinet, with splices safely stored away inside a water-tight cabinet.

Standardized IP68-rated data concentrator cabinets were ordered for all vaults. The cabinets were sized to allow for the data concentrator, fiber termination bulkhead, and Ethernet switch while allowing a roughly 16-inch, square space for a device(s) that may be added later. While this design was full of solid choices made on paper, it was discovered that the cabinets were too large to fit through the manhole entrance in a handful of our older transformer vault roofs. A supplementary order for smaller cabinets (without the space for future devices) gave an option in these circumstances, although in some cases it was decided to change out the roof for a 30-inch manhole opening, which is also desirable for access and rescue needs.

With the fiber pulled in and terminated, the time had come to start talking to network protector relays. The employees needed to commission an Ethernet switch, data concentrator, and network protector include a SCADA technician, PCM/relay technician, system operations desk attendant, and a crew of cablemen — and that’s only if it goes well. When it doesn’t go well, this highly qualified group of employees is mostly idled while troubleshooting occurs.

Commissioning took longer than expected (or maybe more accurately, longer than hoped for). The relatively lengthy timeline is partially due to a design decision to use Avista’s existing SCADA system for issuing control commands and taking data acquisition, instead of taking advantage of the manufacturer-supplied interface inside of the data concentrators (in order to provide familiarity for company operators).

 In the footsteps of the past

Avista communications technician Cory Christie prepares fiber ends for termination. The fusion splicing machine used to apply terminal ends sits ready in the background.

“Old” takes on a new meaning in networks — rather than a 50-year-old substation reaching the end of life, we have an 80-year-old transformer vault. Rather than a 60-year-old pole leaning over on the side of a rural road, we have a 110-year-old brick manhole whose cracks are spreading. The wooden duct lines are gone from our system and the early iron pipe ducts have long since rusted away, but the cable is still there, buried in the rust and dirt.

Similarly, when we look at the implementation of an overarching SCADA system, prior efforts throughout the industry and here at Avista have definitely informed our way of thinking. When SCADA was first implemented at Avista substations, there must have been an uproar because all eight of them were manned 24 hours a day, with on-site housing. Around 100 years later, the idea of somebody in a central operating room opening a network protector is slowly becoming “normal,” but that is not to say that we should take the change lightly.

Downtown networks, which are usually the oldest operating districts of our utilities and the places where power was first distributed over a century ago, are a thing of invention and ingenuity. Why has “Smart Grid” not been applied to some downtown networks in the same way that it was across radial distribution systems over the last decade? Simply put, many of the drivers for Smart Grid on a radial system (reliability and remote restoration for example) do not apply to a downtown network.

Downtown networks were more reliable a century ago than a “Smart” radial system is today. The reason to implement SCADA in a downtown network is to improve safety and efficiency from an operational standpoint and to provide a platform for further innovations. While we cannot yet claim full success with the implementation of our system at Avista, early signs show great promise for the future.