Can anyone withdraw money from my account if they know my account number

Simple Answer is NO!

Your Account Number is just for the sake of Information. Even someone knows your Account number, its IFSC code & your name, they can only deposit money to your account. With all these 2 information, no one can actually take out money from your account.

To take account money from your account, the other person should have access to either of the following:

1. Your Signed Cheque book
2. Your Debit Card with PIN number
3. Your Net banking User Id & Password & also they will need your Mobile phone to authenticate the transaction by entering the OTP.

So stay assured, no one can take out money from the account, with just a Account Number!