Open Sourcing the Outage Management System

I started in the Radio Room back in March (or is it April) 2008 and I was horrified to discover that their logs consists of *gasps* logbooks. Tons and tons of logbooks, around 2 or 3 for every year. Everytime our GM, managers, or supervisors asks for updates to a particular complaint or interruption, we’re forced to flip over pages of unintelligible penmanship and do a follow-up call.

Our workflow is equally mortifying. The ones on a 12MN-8AM shift is obligated to write (yes, write) down all the complaints and interruptions we’ve received. Again, there’s the penmanship issues, it’s like a tropical typhoon on a piece of paper. My colleagues on 8AM-4PM and 4PM-12MN shift would have to write down two copies of job orders. One for the line maintenance crew and another for the 12MN-8AM operator.

Our workstation is a glorified cellphone using My Phone Explorer to manage SMS and Contacts. And there’s the 1000 contacts and 1000 SMS limits so we have to archive the messages every now and then.

Coming from a call center industry, I was used to utilizing our workstations to its full potential. So I proposed we install something similar. My research led me to SugarCRM. It’s an ok application, but we need a way to log interruptions and writing a plugin would take up precious time.

So I started creating a web application from scratch. Since it’ll be accessed from multiple workstations, I decided a web interface’s the way to go. I have no intention nor the patience to hop from one station to another installing and updating the software. I usually just give our IP Address and advise our users to fire up their preferred browser with specific instruction to stay away from IE6 for obvious reasons.

It’s initially called a Case Management System wherein each complaint or request becomes a case and the maintenance crew would work on it. But then ERC released a memo in 2009 that all cooperatives should have an Outage Management System. So I just renamed it from CMS to OMS, heh!

Fast forward to December 2010, I’ve been working on our OMS for almost two years now and the development’s still ongoing.

So Why Open Source the OMS

We’ve had regular visits from ERC and other electric cooperatives. Usually it’s due to our SCADA, and most of the lakbay-aral trips are solely for the purpose of learning how our SCADA works and how they can implement it on their sub-stations.

But just recently, an ERC representative doing audits and another from an electric coop down south (forgive me for forgetting the name of the coop) saw our OMS and was interested in using it themselves. Fortunately I was the one on duty and explained to them that it is not possible at the moment.

Since I’ve been getting immediate and critical feature requests – at most once a week, I was forced to hard code a few crucial items. Bad programming practice, yes, but I did manage to meet most of the deadlines. So it won’t work with other cooperatives since it’ll show up as a BENECO-generated report.

The second and the most important factor for hard coding is so I could minimize database queries and file-based access which is taking a toll on our server-cum-workstation. Our OMS is running on a very modest system composed of 3.0GHz Processor, 1GB RAM, and 80GB hard disk drive.

And lastly, although our department manager and direct supervisor sanctioned this project, the higher management aren’t aware that such program exists. Sure they’ve seen it, but for all they know, it’s just a regular electronic log system. This is also the reason why the program isn’t licensed and is running on a desktop workstation and not on a server.

But because I developed our OMS during company time, if they want the code, the requests should go through official channels.

As such, once I start creating an Open Source version of our OMS (which might be hosted on GitHub), I’ll have to start everything from scratch on my free time to prevent conflict of interests between my employer and my hobby.

Summary of Main Features

Our OMS is not linked to our SCADA. Only authorized applications created by our SCADA engineers area allowed access. But since the SCADA systems’ right beside us, it doesn’t really matter. Besides, our SCADA’s only installed on our sub-station. While we can monitor Sub-station and Feeder interruptions, we cannot monitor everything else. So we have to rely on callers to determine which area has power outage.

BENECOs’ OMS is optimized for monitors with at least 1152px width and designed to be used by Radio Operators (we have 2 landlines, a fax machine, a cellphone, a radio and an internal IP-based phone, which we sometimes have to answer simultaneously) who needs to have ready information on everything at hand. The data presented to regular users (including technical personnels) may prove overwhelming but still insufficient to us.

Everything is loaded via AJAX to preserve bandwidth and you only need to load the whole page upon startup and when accessing a report. When we need to access the complaints module, only the middle pane would load. But sometimes, the three panes isn’t enough, so some modules are loaded in a modal (which I’ll show to you later).

Upon loading, the left-pane either contains the login form or the sub-menu while middle pane show summary of complaints and issues. The right pane contains our notes, directives, or summary of memo to remind Operators.

The summary contains Open Complaints, Streetlights, Interruptions for Updates, and Deferred statistics. It will also list critical issues such as bumped poles (which can be such a bitch), ongoing interruptions, and areas with power outages.

The term No Light & Others are applied to cases where two or more housesholds are without a light caused by tripped transformer or sectionalizer fuses, reclosers, etc.

Upon logging in, the left-pane would load the sub-menu to other modules which could only be accessed by authenticated users.

It allows access to administrative tasks such as updating the crew code, streetlights and protection devices such as Reclosers and Sectionalizer data.

Forms and other information are loaded on either the middle or the right pane. Mostly the Complaints module is exclusive on the middle-pane since we need to have it open all the time.

The Create Complaint form would always show up right above the summary of complaints and can only be invoked when clicking the Create New Complaint from the sub-menu or the plus icon on the complaints menu. The summary of interruptions, in the meantime, lists the 30 most recent interruptions, including scheduled and ongoing power interruptions. Interruptions requiring review would have the warn icon right below the arrow icon, which will show interruption details.

See, I told you it’s too overwhelming for users.

The detailed complaint is a bit tricky. It contains Endorsement, Findings, and Resolution details. It also contains SMS history and call interactions.

Sometimes, complaints spans more than 1 day and attended by two or more maintenance crews. As such, the OMS’ complaints module can handle multiple endorsement, findings, and resolution details. If it needs to be endorsed to another department, we just tag it as deferred. And if the said department re-endorses it to Maintenance, we just re-endorse it to crew. This prevents multiple open Complaints with same issue reported by a same person, we work on a single case at a time.

If a complaint reaches 48 hours, the OMS will flag and compel the Operator to update the information.

From this interface, we can also call the complainant or send or reply to an SMS message (if the cellphone is logged in the Contact No. field).

The complaint details also contains a print report link which will open another tab or window with a pre-formatted page for printing.

If a resolved or closed complaint expires (48Hrs), this will be locked to prevent tampering. But a ‘Clone Complaint’ link will be available which will copy the complaint details into the create new complaint form.

We try to pack as much detail as possible which includes the Operator name, the date and time, as well as remarks or comments if we have one.

The resolution details also included Utilized Materials, this way we can track our inventory.

Streetlights and No Light & Others complaints will show a different Resolution Form. The streetlights complaints will include brand names and Pole/Streetlight IDs while the No Light and Others form includes Protection Device field.

We can also filter the summary of complaints by date or by status. We can use the search field to locate a specific complaint. I intend to use this as a global search field which will also query Interruptions and SMS History.

I tried to make adding an Interruption data as simple as possible. It will present NGCP-supplied substations, our substation, Feeders, and Reclosers. If the protection device isn’t on the list, searching by Pole ID, Protection Device ID, or Areas Affected is also possible.

And lastly, we have the Call and SMS module which are available only to authenticated users. And both are presented in a movable modal. It interacts with My Phone Explorer but I’m currently working on using Gammu instead to manage it.

The Call modal would allow you to either search for a contact in the phone book or directly call a number.

The SMS modal allows us to retrieve messages, as well as create a new one or reply to a message. The incoming sms script will first check the phone book and match the sms senders’ number, if it doesn’t find one, it’ll try to match the number from the Complaints module. This will give us an idea who sent the message. If there are no match, the script will just save the number as is to the database.

We can also review a senders’ SMS history by clicking on the name or number which will open up a balloon or a tool tip containing the most recent messages.

The message list also contains links to creating a new Complaint as well as past/resolved Complaints.

Our OMS contains a lot more features than those listed here, but if I post everything, this’ll be a manual instead of a blog post.

So anyway, I plan on starting this project first quarter next year. The source code will be publicly available and probably licensed under GPLv3. I’ll just post another blog once I’ve started so other interested Electric Cooperatives could participate.

Leave a Reply