Austin's Thoughts

Challenges of DoD AI/ML Path

As the Depart­ment of Defense (DoD) con­tin­ues to explore dif­fer­ent use cas­es for inte­grat­ing Arti­fi­cial Intel­li­gence (AI) into its every­day process­es, it must learn from past lessons where AI deploy­ments failed. One such les­son is the fail­ure of IBM Wat­son deploy­ments. Since its debut on “Jeop­ardy!” IBM Wat­son has been applied to var­i­ous fields, includ­ing health­care, finance, cus­tomer ser­vice, and more, demon­strat­ing its ver­sa­til­i­ty and poten­tial to solve com­plex prob­lems across dif­fer­ent industries.

Data Cleaning

For IBM Wat­son to be effi­cient, it requires clean data to pro­duce pos­i­tive out­puts. Clean data refers to data that has been processed and pre­pared to ensure it is free from errors, incon­sis­ten­cies, and irrel­e­vant infor­ma­tion. Clean data is cru­cial for build­ing accu­rate and reli­able mod­els. How­ev­er, clean data is rare in the amount the DoD pro­duces because it often lacks uni­form metatags, fails to fol­low a par­tic­u­lar for­mat, and is stored in var­i­ous loca­tions like file stor­age or Microsoft Teams environments.

One rea­son IBM Wat­son deploy­ments failed was that the data need­ed to be cleaned, and the DoD need­ed trained data stew­ards who could add con­text to the data results. Devel­op­ing and train­ing good data stew­ards takes time, and at the height of the Glob­al War on Ter­ror­ism (GWOT), no one want­ed to give up their man­pow­er to clean up the data. Data clean­ing is also time-con­sum­ing and often gets del­e­gat­ed to junior enlist­ed per­son­nel who do not have a con­tex­tu­al under­stand­ing of the data.

Data Ownership

The next chal­lenge for a suc­cess­ful IBM Wat­son deploy­ment was get­ting access to the data. With­in a com­bat­ant com­mand, the senior rank­ing com­mand­ing offi­cer is typ­i­cal­ly the data own­er (often del­e­gat­ed to the senior com­mand­ing intel­li­gence offi­cer for respon­si­bil­i­ty, and the senior com­mu­ni­ca­tions offi­cer owns the sys­tems in which the data resides). How­ev­er, it can be chal­leng­ing. The indi­vid­ual pro­gram offices and direc­to­ries that spend their fund­ing to devel­op the sys­tems have con­trac­tu­al own­er­ship over the data.

For exam­ple, sup­pose the Logis­tics Direc­torate devel­ops a pro­gram to keep track of mate­r­i­al with­in their Area of Respon­si­bil­i­ty (AOR), and the direc­torate uti­lizes a Fed­er­al Ser­vice Inte­gra­tor (FSI) to devel­op the pro­gram. In that case, the data is now locked behind sev­er­al lay­ers of bureau­cra­cy. One, if the project is ongo­ing or if the devel­op­ment of the project has stopped, it will require con­tract mod­i­fi­ca­tion to allow for an Appli­ca­tion Pro­gram Inter­face (API) to access the under­ly­ing data or a new con­tract to devel­op new fea­tures with­in a lega­cy code base, which allows for the stat­ed API. Sec­ond­ly, it is often a bur­den­some process to review the lega­cy code base because the orig­i­nal devel­op­ment team, at the time of devel­op­ment, did not com­ment on their code, did not con­sid­er fea­ture sup­port, used pro­pri­etary tools that are no longer sup­port­ed, or com­plete­ly ven­dor-locked in.

Same Problem, New Technology

With the promise of the DoD CIO’s office “Cloud First” approach and its promise of Cloud Ser­vice Providers (CSPs) offer­ing unlim­it­ed access to all the AI good­ness, the DoD is falling into the same trap as it did with IBM Wat­son. For those unfa­mil­iar with the “Cloud First” approach, the DoD CIO’s “Cloud First” strate­gic ini­tia­tive is to mod­ern­ize the Depart­ment of Defense’s IT infra­struc­ture by pri­or­i­tiz­ing adopt­ing cloud com­put­ing tech­nolo­gies. This approach is designed to enhance the agili­ty, effi­cien­cy, and secu­ri­ty of DoD oper­a­tions. As the com­bat­ant com­mands and the ser­vices rushed toward this effort, they often need­ed to real­ize that there were a lot of hid­den costs with the “Cloud First” approach while still not address­ing the issues of data clean­ing and own­er­ship. What mag­ni­fies the issue fur­ther is the lack of spe­cial­ized tal­ent to imple­ment the desired cloud-agnos­tic approach because it requires tal­ent to have in-depth knowl­edge of each CSP to be effective.

“Cloud First” Challenge

The “Cloud First” Approach comes with many hid­den costs. One is the egress cost, or the charges incurred when data is trans­ferred from a cloud provider’s net­work to anoth­er net­work, such as the Inter­net or anoth­er cloud provider. Please note that there was lit­tle to no cost to ingest (data com­ing in) the infor­ma­tion, but the high egress (data going out) cost pre­vent­ed the com­bat­ant com­mands and ser­vices from being cloud-agnostic.

So, the trap is now set. With the promise of AI, the com­bat­ant com­mands and the ser­vices moved their file stor­age to a Cloud Ser­vice Provider (CSP) and, slow­ly, the oth­er parts of their IT infra­struc­ture. Mov­ing the file stor­age was done with lit­tle to no data cleanup, which led to undis­cov­ered data spillage on the exist­ing file stor­age, which now resides on the CSP or data cor­re­la­tion issue. A data cor­re­la­tion issue is where one or more data points can be cor­re­lat­ed to gain more insights. This cor­re­la­tion often clas­si­fies the data at a high­er clas­si­fi­ca­tion. To bypass the issue of data cor­re­la­tion, the com­bat­ant com­mands and ser­vices now over­clas­si­fy the data, which caus­es its own set of issues, pri­mar­i­ly high­er labor costs on per­son­nel to have the nec­es­sary secu­ri­ty clear­ance to access and ana­lyze the data.

Talent Challenge

The DoD fell into the trap of the amount of tal­ent that would be need­ed to main­tain these cloud infra­struc­tures. The DoD went in with the ini­tial thought process­es that they were going to be able to repur­pose the exist­ing tal­ent to main­tain these future sys­tems, but with­out real­iz­ing that there was still an ongo­ing war, which stepped into being able to do some deploy­ment cycles off time that is required to study for these cer­ti­fi­ca­tions. And there were still out­stand­ing sys­tems in place that were required to be able to main­tain while they were try­ing to go to this future cloud environment.

If stat­ed indi­vid­u­als do get trained, have the nec­es­sary cer­ti­fi­ca­tions, and have the secu­ri­ty clear­ance, they will strug­gle with the “Why” state­ment. Why would an indi­vid­ual grade of E‑4 through E‑6 pay or junior offi­cers stay in the ser­vice when their coun­ter­part in the civil­ian sec­tor ini­tial­ly makes between $75,000 to $140,000 with­out the addi­tion­al respon­si­bil­i­ties of serv­ing?  Many make the deci­sion to leave the mil­i­tary, which adds to the con­tin­ued gap of not hav­ing enough cleared per­son­nel with the nec­es­sary clear­ance for the sug­gest­ed AI/ML strat­e­gy or Cloud First Approach with­out hav­ing to sub­con­tract the work.  Along with the his­tor­i­cal­ly low recruit­ing num­bers, the gap con­tin­ues to grow.

Recommendations

The Depart­ment of Defense (DoD) is at a crit­i­cal cross­roads in pur­su­ing AI and ML inte­gra­tion. To nav­i­gate these chal­lenges effec­tive­ly, I pro­pose the fol­low­ing com­pre­hen­sive steps:

1. Mod­ern­ize Con­trac­tu­al Frame­works and Lega­cy Systems
To inte­grate AI and ML effec­tive­ly, the DoD must mod­ern­ize its con­trac­tu­al frame­works and lega­cy sys­tems. This involves con­duct­ing a com­pre­hen­sive review of exist­ing con­trac­tu­al lan­guage to iden­ti­fy mis­sion-crit­i­cal pro­grams. A sys­tem­at­ic process should be ini­ti­at­ed to update lega­cy code bases, pri­or­i­tiz­ing those essen­tial for AI/ML inte­gra­tion. New con­trac­tu­al tem­plates must also be devel­oped to facil­i­tate eas­i­er data access and shar­ing across dif­fer­ent DoD enti­ties, ensur­ing a more stream­lined and effi­cient approach to data man­age­ment and tech­no­log­i­cal advancement.

2. Imple­ment a Data-Cen­tric Approach
A data-cen­tric approach is cru­cial for the DoD’s AI and ML ini­tia­tives. This strat­e­gy begins with estab­lish­ing a robust data gov­er­nance frame­work pri­or­i­tiz­ing data qual­i­ty, secu­ri­ty, and inter­op­er­abil­i­ty across all sys­tems. Com­pre­hen­sive data audits should be con­duct­ed to thor­ough­ly under­stand exist­ing data assets, their qual­i­ty, and their poten­tial val­ue for AI/ML appli­ca­tions. Fur­ther­more, clear cri­te­ria must be devel­oped to guide deci­sions on which data sets should be migrat­ed to the cloud. These cri­te­ria should con­sid­er data sen­si­tiv­i­ty, oper­a­tional impor­tance, and the poten­tial for AI/ML uti­liza­tion, ensur­ing that cloud migra­tion efforts are strate­gic and aligned with the DoD’s broad­er tech­no­log­i­cal goals.

3. Stan­dard­ize Cloud Infrastructure
Stan­dard­iz­ing cloud infra­struc­ture is essen­tial for the DoD to max­i­mize effi­cien­cy and inter­op­er­abil­i­ty in its AI and ML ini­tia­tives. Although con­tracts have been award­ed to mul­ti­ple Cloud Ser­vice Providers (CSPs), focus­ing on a pri­ma­ry plat­form to stream­line oper­a­tions is cru­cial. Ama­zon Web Ser­vices (AWS) is the lead­ing can­di­date due to its high adop­tion rates and robust secu­ri­ty fea­tures. How­ev­er, this deci­sion should not be sta­t­ic; reg­u­lar reassess­ments should be con­duct­ed to ensure the cho­sen plat­form meets the DoD’s evolv­ing needs and keeps pace with tech­no­log­i­cal advance­ments. A com­pre­hen­sive migra­tion strat­e­gy must be devel­oped to facil­i­tate this stan­dard­iza­tion, encom­pass­ing detailed time­lines, resource allo­ca­tion plans, and risk mit­i­ga­tion strate­gies. This approach will ensure a smooth tran­si­tion to a stan­dard­ized cloud infra­struc­ture while min­i­miz­ing dis­rup­tions and secu­ri­ty risks.

4. Invest in Human Cap­i­tal and Knowl­edge Management
Invest­ing in human cap­i­tal and knowl­edge man­age­ment is crit­i­cal for the DoD’s suc­cess­ful imple­men­ta­tion of AI and ML tech­nolo­gies. This involves estab­lish­ing ded­i­cat­ed knowl­edge man­age­ment teams and data stew­ard­ship roles across the orga­ni­za­tion to ensure prop­er over­sight and uti­liza­tion of data assets. Com­pre­hen­sive train­ing pro­grams should be devel­oped to upskill exist­ing per­son­nel in cloud tech­nolo­gies, data man­age­ment, and AI/ML appli­ca­tions, cre­at­ing a work­force capa­ble of lever­ag­ing these advanced tech­nolo­gies effec­tive­ly. To address the chal­lenge of tal­ent reten­tion, the DoD must cre­ate attrac­tive career paths and imple­ment robust reten­tion strate­gies that can com­pete with pri­vate sec­tor oppor­tu­ni­ties. This approach will help build and main­tain a skilled work­force capa­ble of dri­ving the DoD’s tech­no­log­i­cal ini­tia­tives for­ward, ensur­ing that the orga­ni­za­tion can ful­ly cap­i­tal­ize on the poten­tial of AI and ML in its operations.

5. Enhance Resilience and Redundancy
Enhanc­ing resilience and redun­dan­cy is cru­cial for the Depart­ment of Defense (DoD) to ensure unin­ter­rupt­ed data access and com­mu­ni­ca­tion in the face of poten­tial threats to cloud access points (CAPs). The DoD must rec­og­nize that CAPs may become tar­gets, poten­tial­ly dis­rupt­ing crit­i­cal oper­a­tions for com­bat com­mands. To mit­i­gate this risk, robust redun­dan­cy mea­sures should be imple­ment­ed. This can include deploy­ing AWS Snow­ball or sim­i­lar edge com­put­ing solu­tions at com­bat com­mands, enabling con­tin­ued data access and deci­sion-mak­ing capa­bil­i­ties even under Title 10 author­i­ty. Addi­tion­al­ly, the DoD should devel­op and reg­u­lar­ly test con­tin­gency plans to pre­pare for sce­nar­ios where cloud access is com­pro­mised. By imple­ment­ing these mea­sures, the DoD can enhance its abil­i­ty to main­tain oper­a­tional effec­tive­ness and ensure the avail­abil­i­ty of crit­i­cal data and com­mu­ni­ca­tion chan­nels in the face of poten­tial disruptions.

6. Real­is­tic Cost-Ben­e­fit Analysis
A real­is­tic cost-ben­e­fit analy­sis is essen­tial for the DoD’s AI and ML ini­tia­tives, mov­ing beyond the sim­plis­tic view of cloud migra­tion as mere­ly a cost-sav­ing mea­sure. This approach requires thor­ough, long-term cost-ben­e­fit analy­ses that account for often-over­looked expens­es such as data egress fees, ongo­ing train­ing require­ments, and sys­tem updates. These com­pre­hen­sive assess­ments should con­sid­er both imme­di­ate and future finan­cial impli­ca­tions, as well as oper­a­tional ben­e­fits. Addi­tion­al­ly, it’s cru­cial to reg­u­lar­ly eval­u­ate and report on the tan­gi­ble ben­e­fits and return on invest­ment (ROI) of AI/ML ini­tia­tives. This con­tin­u­ous assess­ment process not only jus­ti­fies con­tin­ued invest­ment but also helps iden­ti­fy areas for improve­ment and opti­miza­tion. By adopt­ing this more nuanced and holis­tic approach to finan­cial analy­sis, the DoD can make more informed deci­sions about resource allo­ca­tion and ensure that its AI and ML invest­ments deliv­er real, mea­sur­able val­ue to its oper­a­tions and over­all mission.

7. Align AI/ML Ini­tia­tives with Warfight­er Needs
Align­ing AI/ML ini­tia­tives with warfight­er needs is cru­cial for the DoD to max­i­mize the impact of these tech­nolo­gies on oper­a­tional effec­tive­ness. This align­ment requires estab­lish­ing clear and direct lines of com­mu­ni­ca­tion between AI/ML devel­op­ment teams and front­line mil­i­tary per­son­nel, ensur­ing that tech­no­log­i­cal solu­tions are tai­lored to real-world oper­a­tional require­ments. The DoD should pri­or­i­tize AI/ML projects that direct­ly enhance mis­sion capa­bil­i­ties and oper­a­tional effec­tive­ness, focus­ing resources on ini­tia­tives that pro­vide tan­gi­ble ben­e­fits to warfight­ers. To mea­sure the suc­cess of these efforts, it’s essen­tial to devel­op com­pre­hen­sive met­rics that assess the impact of AI/ML inte­gra­tion on mis­sion suc­cess and warfight­er effec­tive­ness. These met­rics should go beyond tech­ni­cal per­for­mance to include prac­ti­cal out­comes in the field. By main­tain­ing this close align­ment between tech­no­log­i­cal devel­op­ment and oper­a­tional needs, the DoD can ensure that its AI/ML ini­tia­tives deliv­er mean­ing­ful improve­ments to mil­i­tary capa­bil­i­ties and con­tribute sig­nif­i­cant­ly to over­all mis­sion success.

Conclusion

The Depart­ment of Defense’s jour­ney towards effec­tive AI and ML inte­gra­tion is com­plex and mul­ti­fac­eted, requir­ing a com­pre­hen­sive and strate­gic approach. The lessons learned from past fail­ures, such as the IBM Wat­son deploy­ments, high­light the crit­i­cal impor­tance of address­ing fun­da­men­tal issues before rush­ing into new tech­no­log­i­cal initiatives.

The DoD faces sig­nif­i­cant chal­lenges in data clean­ing, data own­er­ship, cloud infra­struc­ture stan­dard­iza­tion, and tal­ent reten­tion. These chal­lenges are com­pound­ed by the rapid pace of tech­no­log­i­cal advance­ment and the com­pet­i­tive land­scape for skilled per­son­nel. How­ev­er, by imple­ment­ing the rec­om­mend­ed steps, the DoD can build a sol­id foun­da­tion for suc­cess­ful AI and ML integration.

The key to this suc­cess is a shift towards a data-cen­tric approach that empha­sizes the impor­tance of clean, acces­si­ble, and prop­er­ly man­aged data. This must be cou­pled with mod­ern­ized con­trac­tu­al frame­works and lega­cy sys­tems that facil­i­tate rather than hin­der tech­no­log­i­cal advance­ment. Stan­dard­iz­ing cloud infra­struc­ture while main­tain­ing flex­i­bil­i­ty for future needs is cru­cial for effi­cien­cy and interoperability.

Invest­ing in human cap­i­tal through com­pre­hen­sive train­ing pro­grams and attrac­tive career paths is essen­tial for build­ing and retain­ing the nec­es­sary tal­ent pool. This invest­ment must be bal­anced with real­is­tic cost-ben­e­fit analy­ses that con­sid­er both the imme­di­ate and long-term impli­ca­tions of AI and ML initiatives.

Per­haps most impor­tant­ly, the DoD must ensure its AI and ML ini­tia­tives align with warfight­er needs. This align­ment will ensure that tech­no­log­i­cal advance­ments trans­late into tan­gi­ble improve­ments in oper­a­tional effec­tive­ness and mis­sion success.

The path for­ward requires patience, strate­gic think­ing, and a will­ing­ness to address sys­temic issues. It demands a bal­ance between embrac­ing cut­ting-edge tech­nolo­gies and main­tain­ing the robust­ness and secu­ri­ty required for mil­i­tary oper­a­tions. By adopt­ing this com­pre­hen­sive approach, the DoD can avoid the pit­falls of past ini­tia­tives and build an AI and ML ecosys­tem that tru­ly enhances its oper­a­tional capabilities.

Ulti­mate­ly, the suc­cess­ful inte­gra­tion of AI and ML tech­nolo­gies has the poten­tial to sig­nif­i­cant­ly enhance the DoD’s abil­i­ty to meet cur­rent and future chal­lenges. How­ev­er, this poten­tial can only be real­ized through care­ful plan­ning, strate­gic imple­men­ta­tion, and a con­tin­u­ous com­mit­ment to learn­ing and adap­ta­tion. As the DoD moves for­ward, it must remain focused on its core mis­sion while lever­ag­ing these tech­nolo­gies to cre­ate a more agile, effi­cient, and effec­tive defense force for the future.

August 14, 2024  Leave a comment