How to Fix a Stuck Cisco Firepower Deployment Notification & How to Delete a Stuck Deployment

Did you ever deploy on your Cisco Firepower/FTD environment and then see something like this?

Let’s start by just deleting a stuck deployment notification, and then I’ll show you how to clear a process on a deployment issue

Deleting a Stuck Deployment Notification

To find the deployment notification that you want to delete, first we’ll log into the FMC and move to root.:

expert

sudo su –

Now we use the OmniQuery.pl tool to query the database and grep for “\ 7\ ” to find the running tasks. Find the uuid of the task that is running, for simplicity I just have the one task listed.To find the notification number:

#OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification;” | grep ” \ 7\  *

Then we make a query that deletes that notification, just like so

To delete the deployment notification:

OmniQuery.pl -db mdb -e ‘update notification set status=13 where status=7;’

or:

#OmniQuery.pl -db mdb -e “delete from notification where uuid-unhex(* (enter notification number here from last command output) *);*

Now if we check again the task is gone!

#OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification;” | grep ” \ 7\  *

The message will clear out after a 5-minute health check, or you can go to Health>Monitor to clear it if you don’t want to wait.

Deleting a Stuck Deployment

To find the deployment process that you want to clear, first we’ll log into the FMC and move to root.:

expert

sudo su –

[email protected]:/var/log# OmniQuery.pl -db mdb -e “select * from notification_status;”

+——–+———–+——-+

| status | label     | level |

+——–+———–+——-+

| 1      | info      | 1     |

| 2      | success   | 1     |

| 3      | normal    | 1     |

| 4      | recovered | 1     |

| 5      | disabled  | 1     |

| 6      | waiting   | 1     |

| 7      | running   | 1     |

| 8      | retrying  | 1     |

| 9      | suspended | 1     |

| 10     | stopped   | 1     |

| 11     | warning   | 2     |

| 12     | critical  | 3     |

| 13     | failure   | 3     |

| 14     | error     | 3     |

+——–+———–+——-+

14 rows in set (0.00045 seconds)

[email protected]:/var/log# OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification where status=7;”

+——–+———————+———————————-+——————————————————————————————————————————————-

| status | category            | hex(uuid)                        | body                                                                                                                                                            |

+——–+———————+———————————-+——————————————————————————————————————————————-

| 7      | deployment:category | 4D595DC11057366AA951161CB9A7BD4B | {“property”:”deployment:deployment_initiated_for_the_device”,”arguments”:{“PHASE”:”Phase-0″,”DEVICE_ID”:”339302418939″,”JOB_ID”:”562640821028″},”literal”:null} |

+——–+———————+———————————-+——————————————————————————————————————————————-

1 row in set (0.000325 seconds)

[email protected]:/var/log# OmniQuery.pl -db mdb -e ‘delete from notification where uuid=unhex(“4D595DC11057366AA951161CB9A7BD4B”);’);’

[email protected]:/var/log# OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification where status=7;”

[email protected]:/var/log# OmniQuery.pl -e “SELECT j.JOB_NAME,CURRENT_STATE,DEVICE_ID,d.DEVICE_STATUS, CREATE_TIME, LAST_UPDATE_TIME, LAST_ACTION_TIME from jobs_table j,jobs_device_status d where j.job_id=d.job_id AND CURRENT_STATE=’Deploying’;”db mdb -e “select status,category,hex(uuid),body from notification where status=7;”

[email protected]:/var/log# OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification where status=7;” ” 

+——–+———-+———–+——+

| status | category | hex(uuid) | body |

0 rows in set (0.00032 seconds)

8 Comments

  1. I think this deletes the ‘notification’ but the process still runs in the background, this is true at least for domain management process

  2. I do the following in 6.5
    sudo OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification;” | grep “| 7 |”

    and then kill the PID
    sudo OmniQuery.pl -db mdb -e ‘delete from notification where uuid=unhex(“pid”);’

  3. Maybe I am old school (or just old) but how is this a “fix”?

    You look for a “running task” by grepping a DB and then you kill the task/process/pid? Shouldn’t this be said to be killing a running task to get rid of a notification telling you it is still running?

    Where is the validation that the task you are being notified about is actually completed before killing the notification? What about prevention of the same issue coming back? Those are fixes.

  4. This doesn’t kill the actual process. It only deletes the task from the Notifications list.
    If the process is still running it will reappear in the Notifications list after a few seconds.

    You can check in the action_queue to see if a certain task is actually still running:
    OmniQuery.pl -db mdb -e “SELECT description,target,create_time,last_state_change,TIMEDIFF (last_state_change,create_time) as Duration ,message,state,hidden FROM action_queue WHERE state=3 order by last_state_change;”

    If the task is still shown in the queue, it’s still running and deleting it from the Notifications list won’t help.
    If it’s not shown in the queue, it just means that it’s a stuck notification which you can then safely delete.

    1. Pascal, that is correct. I wrote this as I had a customer that had a suck process for a year on their firepower
      the codes I listed here, clear that out for them
      This isn’t meant to stop an active deploy or update, just a stuck notification that is annoying as hell

Leave a Reply

Your email address will not be published.