An overflow doesn't necessarily mean invalid, rather it is a carry that can be used for the adjacent byte when doing 16bit math with 8 bit registers.
Consider adding the max 8bit values together which is 255+255 = 510
C REGISTER C = the carry bit (Over flow) and Regiter is the 8 bit register
0 11111111 (0xFF)
0 11111111+ (0xFF)
1 11111110 When you include the carry you have the correct answer of 510 (0x01FE)
The overflow is a carry which can be added to the high byte when doing 16 bit adds with 8 bit registers
So in reality if your adding two 8 bit numbers you really need to store the answer into 2 bytes or 16 bits. The 8 bit numbers also need to have a dummy high byte register that is cleared and is used as a place holder. The addition is then done in steps where the low bytes are added together followed by the high bytes. The addition of the high bytes uses another command called Add With Carry (ADC).
So lets write some psuedo code that does an A + B = C where A and B both equal 255
Even though we are doing 8 bit math we need to have 16bits so we use two 8 bit registers for each variable. Variable A then becomes AH:AL which is short hand for High & Low bytes in adjacent registers. The assembly ADD commands over write one of the registers with the answer we really don't have a need for a third register called C.
A = 255 load the first 255 into A
A = A + 255 add another 255 to A and store the answer back into A
Here is how you would add 255 + 255 using 8 bit registers
//initalize register
CLR AH Clear AH
LDI AL, 255 Load AL with 255
CLR BH
LDI BL, 255
first add the low bytes
ADD AL, BL (AL = 255 + 255 causes the carry to be set)
next add the highbytes with the carry bit using ADC
ADC AH, BH (remeber the high bytes are clear so this is 0 + 0 + CARRY Bit which is 1)
The contents of the AH:AL registers are 00000001:11111110
SUBTRACTION:
If you uC doesn't have SUB commands (SBC - Subtract through carry) then you can do subtraction by adding the 2's compliment of a number
9 - 2 = 7
1001 - 0010 = 0111
2's compliment is obtained in two steps, first invert the bits and second add one.
For this example we are going to use the 2's compliment of two. I am going to work this as if we have 4 bit registers
0010
1101 (inverted / complimentary)
+ 001
1110 (2's compliment) add this to 9
1001
1110+
10111 This time we ignore the over flow
0111 (7) is the answer
INVALID RESULT:
If you do not perform 16bit math with 8bit register than you have to limit your values to 127 so as to not cause a carry bit to be set (overflow) OR you have to test the carry to see if it has been set and if so an over flow has ocured and you invalidate the result. How you handel such an invalidation is up to you, you can branch to an error handler or set the registers to zero or a max value